OSynth: A Longitudinal, Context-Sensitive Evaluation of Meeting Understanding

AI summary

Overview

Research area: Meeting understanding and summarization, synthetic data generation, and benchmark design for natural language processing and speech.

Technical level: Intermediate (readers should be comfortable with ASR/diarization pipelines, LLM prompting, and standard summarization metrics such as WER and LLM-as-judge).

Scope: This paper introduces OSynth, a fully synthetic corpus of 160 simulated organizations containing 570 hours of generated meeting audio plus surrounding workspace artifacts and an exact latent-state reference, and uses it to measure how much meeting-understanding systems improve when given progressively richer longitudinal context.

What This Paper Is About

Real meeting notetakers increasingly have access to context beyond a single transcript — calendar entries, prior meetings, emails, slides, and enrolled speaker voices — but existing meeting corpora were mostly built as standalone sessions with little surrounding material, so nobody can measure at scale whether that context actually helps. The authors build a synthetic alternative: simulated organizations generated top-down from a shared hidden "latent state" (personas, projects, decisions, tasks), so every observable artifact — audio, transcripts, emails, slides, action items — is consistent with a known ground truth. They then use it to test two commercial model stacks under three escalating levels of context.

Key Contributions

1. The OSynth pipeline and dataset: a hierarchical LLM-based generation pipeline producing 160 workspaces, 2,126 meetings, and 570 hours of synthesized audio, each workspace bundled with aligned transcripts, summaries, action items, outlines, emails, shared documents, slide decks, and per-speaker voice enrollments derived from one shared latent state.
2. A realism validation: a two-alternative forced-choice human discrimination study in which annotators identified OSynth excerpts as synthetic 56.3% of the time versus 79.6% for an unrefined control (chance = 50%), plus an adversarial transcript-refinement procedure grounded in real AMI transcripts.
3. A context-sensitive evaluation protocol: three strict-superset context levels (L1 single-meeting audio, L2 plus the system's own prior-meeting outputs, L3 plus full workspace artifacts) scored both with deterministic "isomorphic" metrics computed against the latent state and with standard LLM-as-judge summarization metrics.
4. An empirical demonstration that workspace context yields large gains on entity-sensitive dimensions (action-item assignee accuracy, number recall, technical-term recall, cross-meeting speaker-identity consistency) that standard metrics and prior benchmarks largely fail to expose.

Main Findings

• Synthetic transcripts are hard to distinguish from real ones: across 24 annotators and 432 non-attention trials, OSynth excerpts were correctly flagged 213/378 times (56.3%), versus 43/54 (79.6%) for the unrefined control — a gap of +23.3 percentage points (Fisher two-sided p = 0.001). The authors note 56.3% is still above chance (binomial p = 0.016), so OSynth is detectable, just not obviously so.
• Context helps on entity-sensitive dimensions: for the OpenAI stack, number recall rose from 0.680 (L1) to 0.883 (L3), technical-term recall from 0.684 to 0.778, action-item assignee accuracy from 0.485 to 0.690, and cross-meeting speaker-identity consistency from 0.220 to 0.476.
• The Gemini stack shows the same pattern: number recall 0.960 → 0.975, technical-term recall 0.809 → 0.841, action-item description quality 3.55 → 3.80, and assignee accuracy 0.069 → 0.630, the largest jump in the table.
• WER barely moves: OpenAI WER went from 0.162 (L1) to 0.144 (L3); Gemini from 0.099 to 0.095, with overlapping 95% bootstrap confidence intervals in both cases. The authors argue this shows word error rate absorbs context gains into low-value tokens while the entity-level metrics capture them.
• Standard judge metrics compress the gains: on the OpenAI system, L1→L3 gains reach +0.30 on LLM-judge faithfulness (3.10 → 3.40) and +0.31 on coherence (4.60 → 4.91), versus +0.26 on speaker consistency and +0.20 on both number recall and assignee accuracy in absolute terms on isomorphic metrics.
• One non-monotonic standard cell: Gemini's LLM-judge faithfulness dips slightly at L3 even though every isomorphic dimension improves, which the authors attribute to reference-free protocols underweighting factual precision.
• Enrollments and workspace text are complementary: an OpenAI ablation separates the L3 increment into voice enrollments and workspace text artifacts; workspace text explains almost all of the number-recall gain, while assignee accuracy benefits from both streams.
• Directionally confirmed on real audio: on eight real AMI meetings (series ES2002 and ES2003), the Gemini system's faithfulness rose from 3.50 (L1) to 3.63 (L3), with coherence at the 5.00 ceiling in both settings — a small study the authors treat as convergent validity, not a second measurement.

Methodology in Plain English

The team generated each simulated company top-down in four stages using claude-sonnet-4-6. First, sample an organization: an industry and size, an org chart of named personas with roles and speaking styles, a list of active projects, and a schedule of recurring meetings. Second, fix a multi-week calendar window and write weekly plans describing which projects advance, which decisions get made, and which action items open and close. Third, break those plans into a concrete meeting calendar plus the surrounding artifacts — emails, shared documents, slide decks — each declaring its cross-artifact references so the pipeline can build a dependency graph and generate artifacts in a valid order, accumulating a consistent history of events.

Fourth, generate turn-by-turn dialogue, then push the draft transcripts through an adversarial refinement loop: a gemini-3.1-pro-preview discriminator compares the candidate against a length-matched real AMI excerpt (with speakers relabeled generically so length and names give nothing away) and a claude-opus-4-7 refiner revises it, for up to five iterations. Splitting the judge and refiner across model families was deliberate — a same-family judge was fooled after one iteration in preliminary runs.

Transcripts are rendered in three variants (canonical written, semantic-verbatim with disfluencies, and a TTS form), and the TTS form is converted to multi-speaker audio with the Qwen3-TTS 1.7B model, cloning voices from persona-matched CommonVoice spontaneous-speech clips, then mixed with realistic gaps and passed through room simulation via pyroomacoustics.

For evaluation, two pipelines are compared stage-for-stage: an OpenAI stack (gpt-4o-transcribe-diarize, gpt-audio, gpt-5.5) and a Gemini stack (gemini-3-flash for transcription and summarization, gemini-3.1-pro-preview for enrollment-based speaker ID). Each meeting is scored at L1, L2, and L3 on a held-out set of 238 meetings from 15 workspaces, with 95% bootstrap percentile intervals from 1,000 resamples. Metrics divide into deterministic comparisons against the latent state (numbers, jargon terms, assignees, speaker-identity stability) and LLM-as-judge scores from a fixed judge, claude-sonnet-4-6, re-run with an open-weights judge as a robustness check.

Why This Matters

Impact on research. The paper targets a measurement gap rather than a modeling gap: prior corpora (ICSI at 72 hours, AMI at 100, AliMeeting at 119, ELITR-Minuting at 80, NOTSOFAR-1 at 32, and MeetingBank at 3,579) mostly lack longitudinal workspace artifacts, and several anonymize entities, which makes entity-sensitive evaluation impossible. OSynth supplies 570 hours with known ground truth and no redaction needed because nothing is real. It also revives a long-standing critique that word error rate weakly reflects understanding, by showing a case where WER moves a little while entity accuracy moves several times as far.

Real-world applications:

• Evaluating and debugging deployed meeting notetakers that already have calendar, email, and slide access.
• Testing cross-meeting speaker identification, where a person must map to the same name across an entire series.
• Stress-testing action-item extraction and assignee attribution for project-management integrations.
• Providing safe, shareable data for teams that cannot release or use real internal meeting recordings.

Industry relevance. Because the latent state is known by construction and the pipeline is cheap to re-invoke, fresh held-out splits can be generated as evaluator models evolve — addressing the concern that AMI, ICSI, and MeetingBank transcripts may already sit in modern LLMs' pretraining data. The paper is authored by researchers at Otter.ai, and the release is positioned directly at the production setting of commercial notetaking systems.

Future Directions

• Scaling the AMI anchoring study beyond two series of four meetings each, to test whether the context effect holds up numerically on natural recordings rather than only in direction.
• Improving audio naturalness: the released audio uses a single multi-speaker TTS model with one cloning prompt per persona, and turns are paced more uniformly than the spontaneous-speech prompts, giving a perceptibly read-aloud quality the authors flag as a limitation.
• Adding realistic noise to non-audio artifacts, since workspace emails, slides, and documents are clean by construction with no typos or transcription noise.
• Broadening diversity beyond the workspace random initialization, which the authors identify as a constraint that will bind as the dataset scales; they also report preliminary generation in five additional languages while the released corpus is English-only.

Target Audience

Researchers and engineers working on meeting summarization, speech recognition, diarization, and speaker identification; benchmark designers who need context-rich, releasable evaluation data; and product teams building assistant or notetaking systems who want to quantify the value of longitudinal context before investing in it. Readers focused on purely acoustic or single-turn ASR advances will find less here, since the paper explicitly positions itself downstream of transcription quality.

Authors’ abstract

Meeting notetakers are increasingly deployed in settings where longitudinal context is available including calendar metadata, prior meetings, emails, slides, and speaker enrollments; however, meeting corpora remain limited in terms of size, domain coverage, and available context. We present OSynth, a large-scale synthetic dataset of simulated organizations comprising 570 hours of meeting audio together with associated artifacts and a latent ground-truth state. In a pairwise comparison, human raters distinguish OSynth meetings from real meeting transcripts with only 56% accuracy based on excerpts of 150-300 words. Across state-of-the-art LLMs, additional context yields significant gains on several entity-sensitive dimensions, demonstrating limitations of contemporary approaches and the potential usefulness of OSynth in developing better-contextualized meeting-understanding systems.