Audio Units, Segmentation, Labels, and Timestamps

AudioSet is the largest clip-level audio corpus ever assembled. In 2021 the team that built it went back and measured what its own annotation unit had cost. They re-annotated part of the corpus at roughly 0.1 second resolution. Then they set that against the labels everyone had been training on. The abstract states the arithmetic of the mismatch: “We devised a temporally strong evaluation set (including explicit negatives of varying difficulty) and a small strong-labeled training subset of 67k clips (compared to the original dataset's 1.8M clips labeled at 10 sec resolution).”

The 1.8M clips were not wrong. They simply carried no onset and no offset. Nothing trained on them could produce one, and a single clip could hold several overlapping events under one tag. Fine-tuning on a mix of weak and strong labels raised d' on that strong evaluation set from 1.13 to 1.41 for a ResNet50. Those 67k re-annotated clips are what it cost to put the boundaries back into a corpus collected without them. The bill went to the same organisation that had chosen the unit, years after it chose it.

Nothing failed during training in that story. It failed years earlier, at the moment someone decided ten seconds was the unit.

Annotation defines what a model is allowed to learn. A clip-level tag, a frame label, a time interval, a transcript, a speaker turn and a session outcome each answer a different question. Each brings its own rules for boundaries, overlap and uncertainty. They also cost very different amounts to collect, which is why the question so often gets settled by convenience. The cheap unit is the one already sitting in the archive. And 2,084,320 ten-second YouTube clips are far cheaper per hour than 67k clips re-annotated at 0.1 second resolution. It should be settled instead by what the system will be asked to promise. It should also be settled before the architecture. The architecture can be swapped out in an afternoon. The corpus cannot.

Those 61 categories, and that 4620/1680 split into utterances, serve one task. Change the task and the cut moves. What settles one case settles none of the four workflows below. The only thing they share is that each has to make the choice out loud. Each also has its own way of getting it wrong.

One distinction sits under all four habits, and each of its three kinds has a canonical corpus behind it. A clip-level label says that a property occurs somewhere in a recording and nothing at all about where. Its canonical form is AudioSet, described on its own project page: “AudioSet consists of an expanding ontology of 632 audio event classes and a collection of 2,084,320 human-labeled 10-second sound clips drawn from YouTube videos.” That is 5.8 thousand hours, with 527 classes actually annotated in the released data. Every tag records presence within the ten-second window and nothing about onset or offset.

A strong temporal label carries the where — the 67k AudioSet clips re-annotated at roughly 0.1 second resolution. A sequence annotation carries the order and the content as well. That is what TIMIT's time-aligned orthographic, phonetic and word transcriptions are, and what a lexicon of 61 distinct phonemes is used to write. They cost more in that order. They answer different questions in that order. Each has to be checked in a different way. Reading one as though it were another is how 2,084,320 ten-second tags end up standing behind a claim about when a sound began.

So the boundary policy goes on paper before any annotator starts. It goes on paper because four separate choices are about to be compressed into a single score: what is measured, what is modeled, what the system decides, and how any of it is verified. The DIHARD evaluation plan is what a written policy looks like when it is done in public. No forgiveness collar on the reference segments, overlapping speech evaluated, both stated before any system was submitted. That rule is why DIHARD III could report 35% to 45% DER on meeting speech, web videos and restaurant recordings, instead of a comfortable single number across 11 domains.

A written policy is also the only way to put the ontology itself under test. Annotators can disagree only about labels the scheme already offers them. An agreement figure measures how consistently people apply the categories they were handed. It says nothing about whether those were the right categories — a lexicon of 61 phonemes, an ontology of 632 audio event classes of which 527 are annotated, ten sound classes in MATS. The categories have to be argued with directly, and it has to happen before annotation volume makes them permanent. At 2,084,320 clips they are permanent.

Boundary-tolerant event precision and recall are computed against a tolerance somebody chose. The community that uses them most has written that choice down. The metrics come from a 2016 paper that leaves the collar as a free parameter of the sed_eval toolbox. That paper cites ±100 ms in DCASE 2013 and 250 ms in DCASE 2016, with the offset condition being the same collar or 50% of the reference event duration. DCASE 2017 fixed the parameter and published it: “An event in the system output is considered correctly detected if its temporal position is overlapping with the temporal position of an event with the same label in the ground truth. A tolerance is allowed for the onset and offset (200 ms for onset and 200 ms or half length for offset)”. Its segment-based metrics, separately, score activity on a fixed one-second grid.

That is the whole point. The same system scored on a one-second grid and scored at 200 ms is being asked two different questions. A collar that moved from ±100 ms in 2013 to 250 ms in 2016 to 200 ms in 2017 is not a property of sound. The figures become readable only beside four things: the unit they were computed in, who was labelled and how many of them, how uncertain the number is, and what the recording setup was.

The handbook page is where all of this becomes somebody's job. Write one for a scheme you are using now. Write it so that a colleague could pick it up and challenge a clip-level tag against a finer release requirement without needing you in the room. Three things have to be on the page: what your definition of the ontology assumes, one counterexample the definition handles badly, and what a measurement of how annotators actually behave would then oblige you to change.

The counterexample is the part people skip, and the record shows what skipping it costs. AudioSet reached 2,084,320 clips before anyone measured, in print, that a strong-labeled subset of 67k clips could move d' from 1.13 to 1.41. MATS spent 133 annotators on 3,930 files precisely to keep the disagreements visible. An afternoon spent on the awkward cases is the cheaper version of both.