Speak for Me: Giving LLMs the Situational Awareness to Participate in a Meeting

AI summary

Overview

Research area: Multi-party dialogue and proactive LLM agents, specifically online meeting delegation (an LLM proxy representing an absent participant in a live meeting), evaluated on the AMI Meeting Corpus.

Technical level: Intermediate. The paper is readable without deep reinforcement-learning background, but assumes familiarity with LLM agents, prompting, state tracking, and evaluation metrics such as recall and F1.

Scope: The paper introduces CAPA, a modular perceive–act–recalibrate architecture that maintains an explicit meeting state so an LLM delegate can decide whether, when, and what to contribute on behalf of an absent participant, plus an episode-level evaluation protocol scored against participant-owned idea units.

What This Paper Is About

In online meetings, an LLM asked to stand in for an absent participant mostly stays quiet: a controlled replication on the AMI corpus shows prompt-only delegates remain silent on 51.4% of the absent participant's valid talking opportunities. The core problem is that these agents have no structured way to track who holds the floor, what has already been covered, what remains unresolved, and where each participant stands, so they cannot recognize when to intervene. The paper's goal is to supply that missing structure and to measure intervention quality along three axes: whether, when, and what the delegate contributes.

Key Contributions

1. CAPA architecture. A perceive–act–recalibrate loop over an explicit meeting state, which shifts multi-party LLM delegation from context-reading at decision time to continuous state maintenance (Perceiver, Predictor, Controller with three helpers, Generator, dual judges, Recalibrator, and a four-part Shared Memory).
2. A unified episode-level evaluation protocol. The protocol scores delegate interventions against the participant's actual idea units, capturing both timing (cue and anchor reference points, a k-turn window) and content alignment, with schema-constrained LLM judges validated against human annotations at Cohen's κ = 0.71.
3. Empirical results on the AMI corpus. On 137 meetings, explicit state tracking reduces the silence rate from 51.4% to 2.5% and doubles credited recovery, while hallucination stays at 0.6%.
4. A failure-mode diagnosis. The work shows that state tracking shifts errors from opaque omission to bounded, module-attributable near-misses, stratified into a four-category taxonomy of uncredited attempts.

Main Findings

• Silence collapses, recall doubles. CAPA reduces the silence (not-attempted) rate from 51.4% to 2.5%, raises loose recall from 26.1 to 52.2 and strict recall from 10.7 to 25.1, and raises Decision F1 by 24.9 points (from 38.1 to 63.0) on 137 AMI meetings.
• Grounding is preserved. Hallucination stays at 0.6% and redundancy at 0.0%; off-topic rises by 5.2% per episode against a −48.9% drop in not-attempted episodes, which the authors frame as a coverage–restraint trade-off.
• The bottleneck is the floor-taking decision, not timing. Intervention rates differ by 48.9% (97.5% for CAPA versus 48.6% for the baseline), yet the baseline's lead-time distribution on its few credited matches mirrors CAPA's. Of CAPA's 595 credited matches, 93.8% are anchor-aligned or pre-anchor, with a median of 1.0 turn before the anchor (baseline n=293).
• Outcome redistribution. Over 1,126 idea-unit episodes, CAPA absorbs the 48.9% drop in not-attempted episodes through +26.1% credited and +22.8% uncredited attempts; the 27.1% strict–loose gap reflects interventions arriving off-anchor inside the k = 5 window.
• Errors become diagnosable. CAPA's 45.3% uncredited attempt rate decomposes into four mutually exclusive categories over all 504 uncredited episodes: incomplete match (Generator realization deficit), different proposition (candidate curation), late match (timing via m_t and the speaker scorer), and safety/quality flag. The transcript-only delegate's 51.4% silent abstention produces no decision trace at all.
• Meeting state is the load-bearing component. On a 20-meeting subset, replacing the state and recalibration with a longer 50-turn raw window drops loose recall by 22.4%, strict recall by 14.5%, and Decision F1 by 24.7%, while redundancy increases by 10.7%.
• Recalibration refines selection, not the threshold. Removing only recalibration drops loose recall by 3.9% and raises uncredited attempts by 4.6%, with Decision F1 and grounding statistically unchanged.
• Results transfer across backbones with a model-dependent operating point. Across GPT-4o, Gemini-2.5-Pro, Llama-3.3-70B, and Qwen3.6-27B, Decision F1 stays within 57.9 to 69.2. Gemini gains recall at the cost of off-topic contributions; Qwen abstains more (17.8%, about a third of the transcript-only baseline's 51.4%) while selecting most cleanly (off-topic 3.3%, hallucination 0.0%).
• Cross-corpus transfer holds. On 10 replayed ICSI meetings (real research-group discussions averaging six participants, up to ten), silence stays near zero (1.2% versus 2.5% on AMI), loose recall is higher (73.6% [61.1, 86.0] versus 52.2%), Decision F1 is comparable (64.8% [57.4, 74.9] versus 63.0%), and hallucination stays low (1.8%), with redundancy at 0.0% and off-topic at 4.9%.
• Findings are stable under parameter changes. With the state available, the raw window is no longer load-bearing: across N ∈ {10, 20, 30}, loose recall stays between 50.5% and 56.7% and Decision F1 between 61.1% and 63.1%. Expanding the evaluation window k ∈ {3, 5, 7, 9} raises loose recall from 43.9 to 62.7 without inflating anchor synchrony.

Methodology in Plain English

The authors treat delegation as sequential decision-making under partial observability: the delegate cannot directly see the topic, decisions, open questions, stances, coverage, or floor, so it must maintain an approximate belief about them and act on that belief.

They derive six requirements from documented prompt-only failures — four about task design (strict causality, participant grounding, pre-generation commitment, abstention sensitivity) and two about architecture (explicit state maintenance, state-directed correction) — and build CAPA to satisfy them.

At runtime, a Perceiver reads each newly observed turn plus the previous state and writes an updated state into a centralized Shared Memory organized along two axes (stable versus turn-by-turn, and delegate-supplied versus environment-generated), yielding long-term, session, working, and episodic stores. A Predictor issues a forecast of the next turn that is withheld from the action policy. A two-tier Controller–Generator split separates strategy from wording: the Controller chooses Silent or Speak and commits to a discrete proposition first, using a candidate curator, a coverage evaluator, and a speaker scorer; the Generator then phrases that proposition in the participant's style. After the fact, an Environment Judge compares the withheld forecast against the actual continuation (isolating perception error) and a Delegate Judge scores the action against the same window (isolating action error); a Recalibrator fuses both verdicts into a structured update of the state.

Preparation is decoupled from the runtime loop: each meeting is split so the first 60% builds the participant profile and the final 40% serves as the evaluation segment, preventing leakage. Evaluation is anchored on participant-owned idea units, each instantiating an episode with a cue (the information boundary) and an anchor (when the participant actually stated the idea); outcomes are strict hit, loose hit, uncredited attempt, and not-attempted. Three trained annotators inspected 150 idea units and 150 same-proposition decisions to calibrate the LLM judges.

Why This Matters

Impact on research. The paper argues that the dominant failure of proactive multi-party agents is not generation quality but opportunity recognition, and that raw-context scaling does not fix it — a longer context window does not recover stance, coverage, or floor. It also shows that output-rewriting paradigms such as Reflexion are structurally mismatched to this regime, because they refine drafted utterances while the primary failure leaves no draft to refine. Feedback is instead directed at state maintenance. Finally, it reframes delegation errors as transparent and module-attributable rather than opaque omissions.

Real-world applications.

• Delegating a participant into a live meeting when attendance is impossible, so decisions are not made without the relevant stakeholder present.
• Customer-support handoff, which the authors name as a future direction for state-driven architectures with similar causal information boundaries.
• Any setting where a proxy must commit to a specific proposition before phrasing it, since the commit-before-realize design makes abstention an explicit accept-or-reject decision on a claim.
• Diagnostic use of the architecture itself: uncredited attempts generate a stepwise record that supports error attribution and targeted refinement.

Industry relevance. The paper reports a robustness check across GPT-4o, Gemini-2.5-Pro, Llama-3.3-70B, and Qwen3.6-27B, noting that a new backbone can be calibrated on held-out episodes by tightening printed suitability and restraint constants with no retraining. It also flags ethical constraints for deployment: recurring consent from all participants, risks of impersonation under false pretenses, plausible-deniability speech laundering, and asymmetric advantage of represented over unrepresented participants. CAPA is released under the MIT license as research infrastructure, with AMI distributed under CC BY 4.0.

Future Directions

• Generalizing to other domains with similar causal information boundaries. The authors name customer-support handoff as an explicit future direction.
• Meeting corpora beyond AMI. The evaluation is centered on AMI's scenario portion; ELITR and ICSI lack AMI's continuity without adaptation, and parliamentary or formal institutional datasets enforce rigid turn-taking where spontaneous floor-taking is not structurally possible.
• The coverage–restraint trade-off. Off-topic contributions rise by 5.2% per episode, which the paper calls an open design question for fixed-weight LLM agents.
• Backbone dependence. Performance depends significantly on the underlying language model; models with different reasoning capabilities or smaller context windows may produce less accurate delegation, and the specific safety–coverage operating point is model-dependent.

Target Audience

Researchers and practitioners working on LLM agents, multi-party dialogue systems, meeting assistants, and proactive or proactive-style agent architectures. It is also relevant to teams building delegation or proxy systems where an agent must decide whether to speak at all, and to those interested in evaluation protocols that score timing and content against human idea units rather than turn-by-turn text overlap. Readers with only passing familiarity with sequential decision-making will still follow the architecture and results, though the POMDP framing rewards some background.

Authors’ abstract

In online meeting delegation, LLM agents fail to recognize when to speak. With no structured way to track stances, coverage, and floor, they miss the moments where they should contribute. Prompt-only delegates stay silent on 51.4% of the absent participant's talking opportunities on the AMI corpus. We present CAPA (Collaborative Agent Predictive Architecture), an architecture for online meeting delegation. A Perceiver updates the meeting state from each observed turn. A Predictor forecasts how the conversation will continue. A Controller decides whether to speak and which proposition to surface. A Generator phrases the chosen contribution in the participant's style. Two judges score the forecast and the action against the next observed turn. A Recalibrator updates the meeting state from those verdicts for future decisions. To evaluate online delegation, we introduce an episode-level protocol that scores whether, when, and what a delegate contributes around the participant's actual idea units. The protocol's schema-constrained LLM judges align with human annotations at Cohen's kappa = 0.71. On 137 AMI meetings, CAPA reduces the silence rate from 51.4% to 2.5%, doubles credited recovery (26.1 --> 52.2), and keeps hallucination at 0.6%. The failure mode shifts from omission to selection, with each residual near-miss attributable to a specific module of the architecture. Mechanism ablations identify the meeting state as the lever that closes the recognition gap, where raw-context scaling alone does not.