Audio Tagging, Sound Event Detection, and Acoustic Scene Analysis

Three questions get asked of the same recording and they are not the same question. What was audible in it? When exactly did each thing start and stop? What kind of place was it recorded in?

Audio tagging answers the first and makes no claim at all about the second. A label covers the whole clip and says only that something happened somewhere inside it. Sound event detection answers the second, putting each event on a timeline. Acoustic scene classification answers the third, describing a context rather than an incident.

The three are scored on different evidence, and that is where the trouble starts. A system trained and measured on the first question can look finished right up to the moment production asks it the second. The gap is neither hypothetical nor small. AudioSet is the field's largest labelled collection, and its original release carries labels on its 1.8M training clips at 10 sec resolution only. A tagger trained on it has never once been shown where an event began or ended.

Everything that follows is what happened when teams measured that instead of asserting it. Google Research went back and re-annotated its own dataset at roughly 0.1 sec resolution. Audio Analytic showed that changing only the scoring rule reorders a leaderboard. A group at Tampere University kept the ten scenes and the ten cities and changed nothing but the microphone.

The strongest evidence that a clip label is not an event label was published by the people who published the clip labels. In 2021 Google Research went back to AudioSet and named the limit of its own resource. The original release's 1.8M training clips carry labels only at 10 sec resolution. Nothing in such a label says when a sound began, when it ended, or how many separate events a burst of them should become.

So they collected new ones. About 81k annotations with precise onsets and offsets, at roughly 0.1 sec resolution. 14k of those fall in the evaluation set, leaving up to 67k for training — roughly 4% of the 1.8M weakly-labelled training clips. Their abstract states it plainly: “We devised a temporally-strong evaluation set (including explicit negatives of varying difficulty) and a small strong-labeled training subset of 67k clips (compared to the original dataset's 1.8M clips labeled at 10 sec resolution).”

Then they measured what the re-annotation bought. A ResNet-50 fine-tuned on a mix of weak and strong labels moved from d' 1.13 to d' 1.39 on the strong evaluation set including explicit negatives. The paper splits that 0.26 gain into 0.11 attributable to temporal precision and 0.15 to other factors. The new labels were released as an update to AudioSet.

That is the episode the rest of this lesson keeps coming back to. Not because 0.26 of d' is a large number. Because of what had to happen to get it. The team best placed to know what a 10 sec label contains decided it did not contain boundaries. Then it paid for 67k clips of fresh annotation to obtain them. No amount of clip-level precision and recall on the weak labels would have surfaced that. Those numbers are counted over clips, and the missing thing was inside the clip.

Both the vocabulary and the tolerance, then, are settled before a model is trained, and what they settle is how much a label is allowed to mean. An event label indicates that an audible pattern matched an entry in an ontology. It does not prove the physical source, the cause, the danger, the intent, or the uniqueness of that event. A clip-level system never claims an alarm was sounding. It claims a clip resembled the clips annotated that way.

The deeper version of the gap is named in the field's standard tutorial, published in IEEE Signal Processing Magazine in 2021. It puts it this way: “Sound event detection defines the task at event level, but practical methods solve it at segment level, thus creating a mismatch between the problem and its solution.” The same tutorial fixes the vocabulary this lesson depends on. A weak label applies to the whole recording. A strong label applies to specific segments. Its Figure 8 illustrates the 200 ms collar on both onset and offset — a peer-reviewed source for the tolerance, independent of the challenge page that sets it. It calls the use of collars a significant weakness in event-based evaluation. That is the criticism PSDS was built to answer.

The three tasks differ in precisely this respect. Audio tagging predicts labels for a broad clip, sound event detection adds temporal localization, and acoustic scene classification characterizes an environment or recording context. Different units, different label semantics, different error costs. Multiple sources overlap and scenes change inside a single clip. Whether any of that becomes measurable is not something the model decides. It is decided by whether the ontology and the unit you chose leave anything for a continuous-operation test to look at. Multilabel precision and recall support a decision only when they are counted over the unit the system will really run on.

Charging the operator several times for one event is a unit problem rather than a modelling problem. Unit problems are why four words that sound like near-synonyms are nothing of the kind. Each names a different thing to count, and what an error costs follows from which one was picked.

The way to stop all of this being a matter of opinion is to make the benchmark run the way the system runs. On a continuous benchmark another team can watch weak clip labels being treated as exact event boundaries, and can see the alert stream that follows. On a benchmark of isolated clips they cannot.

One published deployment did all four steps in the open. SONYC is an acoustic sensor network that runs continuously in New York City. As of December 2018 it had 56 sensors, primarily in Greenwich Village plus other locations in Manhattan, Brooklyn and Queens. Together they had gathered the equivalent of 30 years of audio data and more than 60 years of sound-pressure levels and telemetry. Its team described it in Communications of the ACM in February 2019. The problem is not academic. New York's 311 line has logged more than 2.7 million noise-related complaints since 2010, roughly 834 a day.

Step one, long recordings rather than clips: theirs is a continuous stream from fixed sensors, not a folder of isolated examples. Step two, an explicit matching and triggering rule, written down before the results. In an 11-month Washington Square Park study, from May 2016 to April 2017, they triggered on instantaneous SPL exceeding a documented threshold. That threshold was 10 dB above a background computed as a moving average of SPL measurements within a two-hour window, the level the city's noise code defines as indicating potential violations. Step three, convert outputs into an operator's burden: the trigger produced a countable number of detections rather than a per-frame score. Step four, review what the detections actually were: “Our analysis resulted in detection of 324 such events we classified by noise source and determined 76% (246) were related to construction as follows: jackhammering (223), compressor engines (16), metallic banging/scraping (7), and the remainder to non-construction sources, mainly sirens and other traffic noise.”

Note what that review is. It is the ontology-gap and acoustically-similar-event audit, done on real detections. Jackhammering, compressor engines and metallic banging/scraping are the classes that a coarse construction label would have collapsed together.

Note too what the paper is candid about. Their machine-listening models performed robust multi-label classification over 10 common classes of urban sound sources in real time on a laptop, and the authors write that they would “soon adapt them to run under the computational constraints of the Raspberry Pi”. At the time of writing the classification was not running on the sensor at all. Write down what your choice of ontology and unit assumes, find the recording that breaks the assumption, and state what your continuous evaluation does when it meets one.

Start with what a good clip-tagging score actually looks like, because the number is lower than the confidence usually placed on it. The best PANNs system reached mean average precision 0.439 on AudioSet tagging in 2020, against a previous best of 0.392. The paper says it in one line: “Our best PANN system achieves a state-of-the-art mean average precision (mAP) of 0.439 on AudioSet tagging, outperforming the best previous system of 0.392.” That is the headline for clip tagging across 527 classes on the field's largest dataset. At the state of the art it sits below 0.5 mAP.

The point of a portfolio is that the numbers are allowed to disagree. Macro and micro multilabel precision and recall can rise while ontology coverage, unknown rejection, and scene-transition behavior get worse. The decision turns on which of those the product actually runs on. For a system running around the clock that is a rate, not an accuracy. Which is why PSDS, the replacement score proposed at Audio Analytic, is the normalised area under a polyphonic sound detection ROC curve whose x-axis is an effective false-positive rate in events per hour. Its e_max, the maximum eFPR of interest, is set to 100 events/hour in the main experiment and 50 in one reported setting. The x-axis is chosen in the units the operator lives in.

The portfolio also has to mark the point where clip labels can no longer stand in for event boundaries. That is where the system should abstain or fall back instead of continuing to fire.