Audio Augmentation, Mixing, Room Simulation, and Leakage

Three different things get called augmentation, and only one of them is safe by default. A label-preserving transform changes the nuisance conditions around the target — the room, the background noise, the channel the audio travelled through. Whatever the label points at stays intact. A target-altering transform changes the labelled thing itself. Stretch a recording in time and the word is still spoken, but the timestamp that was supposed to locate it no longer lands on it. A simulation domain does not modify a recording at all. It manufactures one. Whether the result resembles anything a microphone would have captured is a separate question. Nothing in the pipeline forces anyone to answer it.

That third column is a question rather than a verdict, and it has been answered once, with numbers. In 2017 Ko, Peddinti and three colleagues set training on real room impulse responses against training on simulated ones. The test was the ASpIRE dev set, with roughly 5100 hours of training data. A TDNN trained with real reverberation plus isotropic noise scored 27.8% WER. Simulated RIRs plus point-source noise scored 27.0%. A BLSTM scored 25.7% against 24.6%. In both pairs the simulator was not merely tolerable. It was ahead by a fraction of a point. Their own summary of why is narrow, and worth reading as written: “We find that the performance gap between using simulated and real RIRs can be eliminated when point-source noises are added.” Eliminated — under that condition, on that set, at that scale. The verdict travels no further than the measurement did.

The three transform types usually arrive in the same recipe, defended with the same sentence: the augmented model scored better. None of them inherits its justification from the others. The rest of this lesson is about what happens when that sentence is the only defence on offer.

The cost of that single defence is easiest to see where somebody ran the comparison both ways. One classifier, one corpus, two ways of cutting the split. Seventeen points of accuracy hung on the cut. Arthur Flexer ran it in 2007. The corpus was the ISMIR 2004 contest database: 729 songs by 128 artists in 6 genres. He changed nothing about the model. He changed only where the split fell. Under random 10-fold cross-validation the classifier scored 75.72% ± 3.35. Then an artist filter held, so that all of an artist's songs stayed on one side of the split. The same classifier on the same songs scored 58.50% ± 10.29. Seventeen points of the original accuracy were not genre recognition at all. They were the model recognising an artist it had already heard. Flexer reports an earlier collection where the same filter took accuracy from 71% down to 27%.

The damage does not stop at the headline number, and this is the part worth carrying through the whole lesson. “We show that this does not only lead to over-optimistic accuracy results but also selectively favours particular classification approaches,” Flexer writes in the abstract. The leak is not a constant added to everyone's score. It rewards whichever method is best at picking up the shared fingerprint. So the ranking of methods flips too. A team choosing between recipes on a contaminated split can pick the wrong one while believing it picked the best one.

Notice what the standard check would have reported here. A clean-versus-augmented comparison sets a model against its own twin, and both twins sit on the same split. Under the random split, both of Flexer's configurations would have been measured against a test set that already contained their artists. The gap such a comparison reports is a genuine measurement of something. It is simply not a measurement of generalisation to material the model has never met. Every later idea in this lesson either explains how the 17 points got in or would have exposed them.

That classifier was not badly built. It was evaluated without a stated claim, and a claim is the only thing a result can be checked against. Augmentation changes the training distribution in order to encourage selected invariances or coverage. Every transform in it quietly assumes four things: that the label survives the change, that the result could physically have happened, that it interacts sanely with the sensor path, and that the same assets — the same rooms, noises, speakers, artists — are not also present in evaluation. Flexer's random split failed only the fourth. 75.72% under random cross-validation, 58.50% once the artist filter held.

The operational boundary matters as much as the assumptions. Augmenting more does not automatically make a model more robust. A transform can erase the very thing the target label points at. It can build mixtures no physical arrangement produces. It can overweight easy synthetic conditions until they crowd out hard real ones. It can paper over a defect in how the audio was collected that should have been fixed at the source. The defensible version of the practice states the intended invariance before the transform is applied, runs both clean and stressed evaluation afterwards, and names who owns each of those two ends.

Everything above rests on four terms, and an augmentation report written without pinning them down cannot be read at all. Blur two of them and the proof owed, the unit it is owed in, and the right to decide can all shift without anyone noticing that they moved.

The workflow that would have caught all of this is four steps long, and they are ordered for a reason. Stating the intended invariance comes first, because it is the claim, and everything downstream spends it. Defining plausible parameter ranges comes second. The word doing the work there is plausible: ranges taken from deployment measurements rather than from arbitrary extremes, so the stressed condition is one the system will actually meet. Separating the augmentation assets comes third. That is the step that would have kept 128 artists from appearing on both sides of a split. Running clean and stressed evaluation comes last. It is where a team finds out whether the claim survived the workflow it was carried through.

Run the four in that order and a leak has somewhere to surface. Flexer's invariance, written down, would have read: the classifier should recognise a genre regardless of which artist performs it. Written down, a test set that shares artists with the training set stops being an acceptable place to check that. Before any number is produced, rather than seventeen points afterwards.

That last step needs more than one number, and the one number it must not be alone is the clean-versus-augmented performance tradeoff. It belongs beside three others: robustness measured on held-out real conditions rather than on more synthetic ones, difficult slices where speech was mixed at a level or position the room could not produce, and the coverage and dependence of the augmentation sources themselves.

The first of those three has a measured price for skipping it. In 2017 Vincent, Watanabe and three colleagues ran the CHiME-3 baselines on paired real and simulated recordings. On the development sets the two agreed closely enough to be reassuring: the DNN acoustic model scored 16.13% WER on real dev against 14.30% on simulated dev. On test they came apart. 33.43% on real against 21.51% on simulated. Nearly twelve points of apparent quality existed only in the simulation. No quantity of additional synthetic evaluation would have found them, because more synthetic data is more of the 21.51%. Their own conclusion is not that simulation is useless, and it should be quoted rather than paraphrased: “We show that, with the exception of minimum variance distortionless response (MVDR) beamforming, most algorithms perform consistently on real and simulated data and can benefit from training on simulated data.” Train on it. Do not accept its verdict on itself.

The fourth measure is the one that names the file. A model gaining points from assets it has already been trained on is invisible in a tradeoff and obvious in a dependence report. Together the four stop any single number from carrying a release claim by itself.

All of this fits on a page, and writing that page is the practice. Every augmentation in the contract carries a declared invariance. The contract is only worth having if another team can pick it up and use it to challenge a mixture the room could not have produced. For each transform, write down what its stated invariance assumes, one counterexample to that assumption, and what the clean and stressed evaluation then showed.

The asset audit is the step people treat as abstract, and it is not. There is a specific file. OpenSLR resource 28 ships as rirs_noises.zip: 1.3 GB under Apache 2.0, at 16 kHz and 16-bit. Its description page calls it “A database of simulated and real room impulse responses, isotropic and point-source noises.” It is assembled from the RWCP sound scene database, the 2014 REVERB challenge database, the Aachen impulse response database, and point-source noises drawn from MUSAN. One named, versioned, public archive — and it is what both a training recipe and a test-set builder will reach for without discussing it. That is the thing to hash: not the concept of a room, but the file, per partition, by name.

Had such a page existed for Flexer's classifier, the entry for the split would have had to say which artists. The counterexample would have written itself.