Field Notes — The consent your employees never gave was set for them

For two days, a stranger’s public photographs became composable by @-mention. The rollback removed the button, not the premise underneath it.

For roughly forty-eight hours in July, an Instagram user could @-mention someone else’s public account inside Meta’s newest image generator and pull that person’s posted content into a synthetic image. The person referenced did not have to agree. They had been agreed for: the feature shipped switched on for every account, with opting out left as the user’s homework. After the backlash, Meta withdrew the option two days later, saying the feature had missed the mark. The button is gone. The AI impersonation risk it briefly made visible is not, because the button was never the interesting part.

Most of the coverage read this as a governance story.
Ship first, apologise second, repeat.
That reading is fair, and it is also the smaller of the two stories available here.

The larger one is about where consent now lives. For two days, the decision about whether a person’s face, home, wardrobe, office background and holiday photographs could be fed into a generative model was made at the platform layer rather than by the person. Talent agencies understood this immediately, Variety reported that several told their clients to opt out, citing identity theft. Their clients are professionally aware that a likeness is inventory. Almost nobody else is.

But the same arithmetic applies to people who have never thought of themselves as inventory. The CFO who posts a photo from a conference stage. The IT manager whose feed shows the badge lanyard, the office kitchen, the branded hoodie, the two colleagues he is standing between. The helpdesk lead who tagged her employer in a public post about her first week, for an attacker building a pretext, the scarce ingredient has never been an email address. It is credible context, the texture that makes a fabricated situation feel like a remembered one.

What used to slow attackers down was labour, not law

Nothing about that content was private before July. It was public, scrapeable, indexable, and it has been the raw material of OSINT-driven targeting for years. What changed for those two days was not access. It was cost.

Assembling a convincing visual pretext from a person’s public life is work. Someone has to collect the images, understand what they show, decide which details are load-bearing, and compose them into something a colleague would not question. That work is exactly the friction defenders have quietly relied on, without ever writing it into a control. A feature that turns collection, comprehension and composition into an @-mention does not add a new capability to the attacker’s toolkit. It removes the labour cost of an old one, and labour cost is the only tax most attackers have ever actually paid.

This is the part the rollback does not touch: the model still exists and the content is still public. The composition is still possible for anyone willing to spend an afternoon instead of a keystroke. What was withdrawn was convenience, and convenience is not a security control, even though a great deal of our security depends on its absence.

There is a second thing the rollback does not touch, and I think it is the more durable one. The setting was on by default. Not offered, not prompted, not staged behind a consent screen for the referenced party.. inherited. That default reveals the operating assumption behind the product decision far more honestly than the apology does, and defaults are the most reliable predictor of what the next feature will look like. Two days is also, worth noting, longer than the average phishing campaign stays live.

The consequences nobody is pricing (geeeez)

For defenders, the uncomfortable adjustment is that employees’ public social presence is still filed under communications and reputation, not under attack surface. Threat models enumerate corporate identity assets meticulously (domains, mail flow, SSO, tokens, brand likeness) and then stop at the perimeter of what the company owns. The AI impersonation risk sits precisely outside that line, in accounts the company does not control, on platforms whose product roadmaps the company will never see. Awareness material, meanwhile, still teaches people to look for tells: the extra finger, the waxy skin, the light that comes from nowhere. That heuristic was priced against amateur output. It is being asked to hold against production-grade generation, and nobody has told the people relying on it that the terms changed.

For regulators, this is where the vocabulary starts to creak.
DORA and NIS2 speak fluently about ICT third-party risk, supplier concentration, and critical service dependencies. A consumer photo-sharing platform where employees post their weekends is not a supplier of anything to a regulated firm. Yet a single default-on change to that platform can shift the pretext quality of every social engineering campaign aimed at that firm’s staff, and no article number covers it. The EU’s instinct here has been the awkward, unloved one, the Commission has previously forced Meta to delay AI rollouts pending assessment, and the reporting notes fines exceeding a billion dollars a year over the past three years, but even that posture is built around data usage and harm to users, not around the resulting supply of attack material.

And for organisations, the mispricing is quiet. Likeness abuse appears in a budget, if it appears at all, as a deepfake-CEO line item, usually mitigated by a payment control: call back on a known number, second approver over a threshold. That control assumes the attack arrives at the treasury. The cheaper version arrives everywhere else.. at onboarding, at the service desk, at the vendor who receives a plausible photograph of a person they have never met but have seen before.

The feature is gone. The default that produced it is not, and defaults are institutional, not accidental. In two days, we saw a working impersonation pipeline stood up at consumer scale, opted-in by inheritance, and taken down by embarrassment rather than by design review.

I keep coming back to the timing.
Two days is not an outage.
Two days is a campaign window.