Text-to-Audio and Music Generation

MusicLM was trained on five million audio clips — 280k hours of music at 24 kHz. Google published it in January 2023, and the paper did not stop at how the output sounded. It ran a separate memorization study on the semantic-token stage. It reported a number: “We observe that the fraction of exact matches always remains very small (<0.2%), even when using a 10 second prompt to generate a continuation of 5 seconds.”

Look at what that sentence is made of. A rate. At a stated prompt length, for a stated continuation length, measured at one named stage of the pipeline. None of it comes from listening. The same paper released a different instrument for a different question. MusicCaps is an evaluation set of 5,521 ten-second music clips, each labelled with an aspect list and a free-text caption written by musicians. That one asks whether the model did what the text asked.

Two questions, two instruments, two numbers. A panel rating quality and prompt adherence against 5,521 captions has been given no way to produce an exact-match rate. The 0.2% figure has nothing to say about whether the audio was any good. Neither instrument touches the third question: whether anyone holds the rights to what came out. That one leaves the lab entirely.

Adherence, originality risk and provenance are three separate things to evidence, and none of them is settled by how the audio sounds. Text-to-audio and music models generate waveforms or codec tokens, conditioned on language and other controls.

That sentence hides how specific the choices underneath it are. Meta's MusicGen is a single-stage transformer LM over EnCodec tokens for 32 kHz audio. A stride of 640 gives a 50 Hz frame rate. RVQ with four quantizers of codebook size 2048 gives the token vocabulary. The training set is 20K hours of licensed music. The models come at 300M, 1.5B and 3.3B parameters. Meta's own paper, published in 2023, puts the architectural decision plainly in its abstract: “Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling.”

Every figure in that paragraph is a trade being made. Autoregressive, diffusion and hybrid pipelines reach an output by different routes. They trade fidelity, duration, controllability, structure, latency and editing ability against one another. Not one of those trades is the question Universal, Sony and Warner litigated. And note which fact about MusicGen is not an architecture choice at all: that its 20K hours are licensed. That sits on a different record from the frame rate and the codebook size, and no amount of tuning the other numbers produces it.

Audio that sounds plausible does not prove the model followed the prompt. Nor that the result is original, that the rights were cleared, that the physics is real, or that an edit preserved what it was meant to preserve. Long-range musical form and exact temporal control stay separate challenges from short-clip quality. The hard part, then, is not the model. It is whether the generation contract, written up front, keeps the things evaluation and governance will later ask for. Ratings of quality and prompt adherence can only support a decision that contract already made room for.

The contract is therefore the piece to get right, and it is written first, before anyone chooses a representation and a model. It is where seeds, negative prompts and edit boundaries are fixed. It is where a rule about what a style prompt may name would have gone. Choosing the representation and model comes next — 32 kHz codec tokens at 50 Hz, or a 21.5 Hz continuous latent, or an editing pipeline over an existing recording. That step spends whatever the contract assumed. Evaluate and govern comes last, and it is where anyone finds out what was assumed.

That last step no longer has an open-ended deadline. The EU AI Act entered into force on 1 August 2024. Its Article 50(2) requires providers of AI systems generating synthetic audio to mark outputs in a machine-readable format, detectable as artificially generated or manipulated. The transparency obligations of Article 50 apply from 2 August 2026. A contract that never mentioned marking still arrives at step 4 owing it. By then the pipeline is built and the outputs are already in the world.

Guarding against those four means knowing which route your system took to the audio. The published evidence follows the route, and it does not travel.

The autoregressive route has MusicGen's record: EnCodec tokens for 32 kHz audio, four quantizers of codebook size 2048 at a 50 Hz frame rate, 20K licensed hours, 300M to 3.3B parameters, one transformer LM instead of a cascade.

The diffusion route has its own, in different units. Stability AI's long-form music model, published in 2024, is a diffusion-transformer over a 21.5 Hz continuous latent. It generates up to 4m 45s of 44.1 kHz stereo audio. Its memorization audit is a procedure, not a rating. Identify the repeated audios in the training set — 5,566 of them. Generate from those 5,566 in-distribution prompts, plus 586 out-of-distribution prompts from the Song Describer Dataset (no-singing subset). Rank the generations by closeness to the training data, and listen to the closest 50. The result is stated as plainly as the method: “We extensively listened to potential memorization candidates, and could not find memorization.”

Source-conditioned editing is the third route. It transforms selected regions while attempting to preserve specified original content. It owes a third kind of evidence — preservation outside the requested region — that neither of the two audits above was designed to produce.

Put the two published audits side by side and the point makes itself. MusicLM reports a rate below 0.2%, at one prompt length, on one stage. The long-form diffusion model reports a listening pass over generations from 5,566 repeated training audios and 586 out-of-distribution prompts, and it found nothing. Neither number is transferable. A memorization result earned at 24 kHz on semantic tokens establishes nothing about a 21.5 Hz latent generating 4m 45s in stereo. A structure test on a long diffusion output establishes nothing about a 50 Hz token stream.

Owing evidence is easy to agree to and easy to postpone, so make it a pack. A red-team pack is the material another team needs in order to show that a prompt label pointed the model at someone real, or that a phrase came out of the training set. Two of the papers above have already published theirs, in enough detail to copy.

MusicLM's is a rate with its conditions attached: exact matches, on the semantic-token stage, with the prompt length (10 seconds) and continuation length (5 seconds) stated next to the figure. Stability AI's long-form audit is a ranking-and-listening protocol with its counts attached: 5,566 repeated training audios, 5,566 in-distribution prompts, 586 out-of-distribution prompts from the Song Describer Dataset, top-50 generations listened to. MiRA is the third shape: five similarity metrics, model-independent, with its detection floor above 10% replicated proportion declared in the abstract rather than discovered later. None of the three is a listening panel. All three can be re-run by someone who does not trust you.

Write down what defining the generation contract assumes. Then one counterexample. A style prompt naming a living artist is the obvious one, and since 1 July 2024 Tennessee has defined the exposure it creates. A track carrying a phrase from a released record is the other, and MiRA's floor tells you in advance what a track-level score will and will not resolve. Then write down what evaluating and governing actually does about each.

None of that reaches a release decision unless it is reported, and one number will not carry it. For generated music that one number usually has a name: Fréchet Audio Distance. Gui and three colleagues took it apart in 2024. Its correlation with perceptual quality, they note, is understudied. The score itself is fragile: “We show that FAD performance may be hampered by sample size bias, poor choice of audio embeddings, or the use of biased or low-quality reference sets.” Their proposal is a per-song FAD, so that outlier samples surface instead of dissolving into an aggregate. It is the same shape as MiRA's 10% floor, and the same shape as a memorized bar inside a finished track. Aggregation is where the individual bad case goes to hide.

Provenance, meanwhile, has stopped being a metric anyone chooses. Article 50(2) of the AI Act reads: “Providers of AI systems, including general-purpose AI systems, generating synthetic audio, image, video or text content, shall ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated.” The regulation entered into force on 1 August 2024, and the Article 50 transparency obligations apply from 2 August 2026. That turns provenance coverage from a number on a dashboard into a compliance figure with a deadline.

So the report has a shape. Human quality and prompt-adherence judgments cover a short clip. Temporal event, structure and edit-preservation tests cover what happens over 4m 45s. Hard slices built from prompts that name a real artist or voice cover the exposure the ELVIS Act describes. Figures for latency, controllability, provenance coverage and misuse incidents cover the working life of the system. Drop any of them and short-clip ratings carry the release claim alone — which is exactly the claim that neither the complaints of 24 June 2024 nor Article 50(2) will accept.