Dual Use, Misuse, and Societal Externalities

A dual-use review examines capabilities that support legitimate work while also lowering the skill, cost, time, or coordination a harmful action needs. Societal risk then emerges through scale, diffusion, interaction, and institutional fragility rather than through a single user request. So the review names the threat actors, what they can already do, how they get access, and which model and product affordances they can reach. It then looks at distribution channels, monitoring, downstream dependencies, and defensive capacity. A refusal policy at the interface is one of those layers. It is not the review.

Voice cloning is the capability this lesson follows. The public record on it is unusually complete. Two days before the New Hampshire presidential primary of 23 January 2024, roughly 10,000 robocalls went to residents the callers believed were likely Democratic voters. They carried an AI-cloned voice of President Biden and a spoofed caller ID. An interface review inspects the model, the request, the terms. It does not inspect the call volume, the spoofed identity, the primary two days away, or the people who had to judge in the moment whether the voice was real. Nobody measured that second list before release. Verification was the part that had to absorb the result.

Two days before the New Hampshire presidential primary of 23 January 2024, robocalls carrying an AI-cloned voice of President Biden and a spoofed caller ID went to likely Democratic voters. A federal judge in New Hampshire later put the facts in one sentence: "In January of 2024, two days before the New Hampshire Presidential Primary Election, defendants directed approximately 10,000 robocalls to New Hampshire residents they believed were likely Democratic voters." That order, dated 26 March 2025, let the civil claims in League of Women Voters of New Hampshire v. Kramer proceed.

Two forums acted on the same calls. The FCC imposed a forfeiture on Steve Kramer under the Truth in Caller ID Act: "We impose a penalty of $6,000,000 against Steve Kramer". The federal district court in New Hampshire handled the civil side. Neither forum is the vendor of a model. Neither remedy arrived before the primary.

Capability, accessibility, intent and actor, target and dependency, countermeasure ecosystem: harm needs all five. A countermeasure that touches only one of them tends to move the problem rather than remove it. Two of the five links have been measured directly. Those measurements are what turn this diagram from a checklist into an argument.

Target and dependency decides how much verification a human audience can absorb. People are worse at it than a launch plan assumes. A listening study in PLOS ONE tested 529 participants in English and Mandarin, and its title is the finding: Warning: Humans cannot reliably detect speech deepfakes. The abstract gives the number. "Listeners only correctly spotted the deepfakes 73% of the time, and there was no difference in detectability between the two languages." Playing listeners examples of deepfakes beforehand improved detection only slightly. That 73% is the capacity a plan which says the public will notice is quietly assuming.

The countermeasure ecosystem link has a sized artefact of its own. ASVspoof 5 is the public spoofing- and deepfake-detection benchmark, and its own abstract says what changed: "We introduce the ASVspoof 5 database which is generated in a crowdsourced fashion from data collected in diverse acoustic conditions (cf. studio-quality data for earlier ASVspoof databases) and from ~2,000 speakers (cf. ~100 earlier)." It holds attacks generated with 32 different algorithms across seven speaker-disjoint partitions. An auxiliary set adds a further 30,000 speakers. Challenge participants used it in 2024.

Read the two together and the shape of the harm chain is clear. Capability, accessibility, and intent and actor describe what an attacker can reach. Target and dependency was measured at 529 listeners and 73%. Countermeasure ecosystem is a benchmark with 32 attack algorithms and seven speaker-disjoint partitions behind it — machine detection that exists and is sized. That is not the same thing as a person on a phone line deciding whether a familiar voice is real.

Content moderation removes outputs. Capability restriction removes functions. Ecosystem resilience works on the target instead of the tool.

Content moderation acts at the application layer. It can reduce casual misuse. It needs consistent enforcement and an appeal route. It is vulnerable to evasion and to external tools, because nothing it does reaches material that has already left the product.

Capability restriction acts closer to the harmful affordance, and its clearest example is a regulator's ruling rather than a policy page. The Telephone Consumer Protection Act already restricted artificial or prerecorded voices. On 8 February 2024 the Federal Communications Commission read that restriction onto a new capability: "In this Declaratory Ruling, we confirm that the TCPA's restrictions on the use of "artificial or prerecorded voice" encompass current AI technologies that generate human voices." Calls using such voices therefore need the called party's prior express consent, absent an emergency purpose or exemption. Nothing there rules on the content of any particular call. It rules on the function. Restriction of this kind may cost legitimate utility and requires threat-specific evidence; on the product side it looks like rate limits, identity controls, and disabled cloning.

Ecosystem resilience strengthens targets and verification channels — provenance, authentication, response, literacy. It reduces harm even when content escapes, which is the property the other two lack. It also requires coordination beyond any one vendor, which is why it is the layer most often left out of a launch review. For societal externalities it is not optional.

Asking what users would switch to decides whether restricting a feature reduces harm or merely relocates it. It is also the drill teams most want to skip. Each of these four is a rehearsal for a number that gets produced either way — here, or later, by a court order or an agency ruling after the fact.

Begin from what the system makes cheaper, faster or more convincing. Motive follows capability, and the access paths decide who ever gets to use it. So identify the harmful affordances first — what becomes cheaper, faster, more convincing, or scalable. Then model actors and targets, with realistic motives, resources, channels, and vulnerable institutions. Then test the access paths: API automation, customization, evasion, resale, and open release.

The fourth step, layering mitigations, has already been written down for this exact capability. A joint cybersecurity information sheet from the NSA, FBI and CISA, Contextualizing Deepfake Threats to Organizations, appeared in September 2023. Its executive summary states: "The most substantial threats from the abuse of synthetic media include techniques that threaten an organization's brand, impersonate leaders and financial officers, and use fraudulent communications to enable access to an organization's networks, communications, and sensitive information." The recommendations are specific: media-provenance and real-time verification capabilities, personnel training, rehearsed response plans, and participation in consortiums such as the Coalition for Content Provenance and Authenticity and Project Origin. Note where those land. Training and rehearsal sit inside the target organisation. Provenance and the consortiums sit between organisations. None of the four is a filter on a model's output.

The fifth step reviews externalities: incidents, substitution, public burden, and changes in the threat landscape.

Dual-use risk moves as the surrounding ecosystem moves. A register written at launch is already out of date by the first external incident.

The demonstration arrived in two instalments. The first was a warning from the FBI's Internet Crime Complaint Center on 15 May 2025: "Since April 2025, malicious actors have impersonated senior US officials to target individuals, many of whom are current or former senior US federal or state government officials and their contacts." The methods were text and AI-generated voice messages — smishing and vishing — used to reach those officials' contacts.

The follow-up, on 19 December 2025, had to widen the picture in both directions. It dated the activity back to 2023. It extended the targets to state government, the White House, Cabinet-level officials and members of Congress. Same office, same campaign, an earlier start date and a larger set of targets. A register written before either announcement would have had to be reopened twice.

So name the incident patterns that would force the team to redesign, restrict, remedy, or retire the system. Write them down before the first of them arrives from outside.