Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis

AI summary

Overview

Research area: Medical audio AI, specifically multimodal deep learning for respiratory sound analysis (computer audition applied to pulmonary auscultation), combining agentic orchestration, generative audio synthesis, and clinical diagnosis.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, attention mechanisms, flow matching/diffusion models, audio tokenization, and LLM-based agents. The conceptual framing is accessible, but the implementation details are research-grade.

Scope: The paper introduces Resp-Agent, a closed-loop multi-agent system that unifies controllable respiratory sound generation with multimodal disease diagnosis, supported by a new 229k-recording benchmark corpus.

What This Paper Is About

Automated analysis of breathing sounds (auscultation) could help screen and monitor lung disease, but current deep learning systems suffer from two structural weaknesses: they discard clinically meaningful acoustic detail by converting raw audio into spectrograms, and they are trained on small, heavily imbalanced datasets covering only a few conditions. The authors build a system that addresses both by pairing a generator that synthesizes realistic, disease-specific breathing sounds with a diagnostic model that reads both audio and clinical text, coordinated by an AI planner that decides which synthetic cases to create based on where the diagnostic model is currently failing.

Key Contributions

1. Resp-229k benchmark. A corpus of 229,101 quality-controlled respiratory recordings (roughly 408 hours) spanning 16 diagnostic categories, each paired with an LLM-generated clinical narrative distilled from source metadata. It uses source-disjoint splits (train on ICBHI, SPRSound, UK COVID-19; test on unseen KAUH and COUGHVID) to test genuine cross-institution, cross-device generalization.

2. Resp-MLLM controllable generator. To the authors' knowledge, the first multimodal large language model trained with aligned text-audio supervision for controllable respiratory sound synthesis. It adapts a small text-only LLM (Qwen3-0.6B) via modality injection to predict discrete acoustic units conditioned on both a diagnosis prompt and a reference style, then reconstructs waveforms with a conditional flow-matching decoder.

3. Modality-weaving Diagnoser. A diagnostic model that interleaves clinical text tokens with audio features in a single sequence from the first layer, rather than fusing them late. A sparse set of "audio anchors" acts as global attention hubs spaced roughly 80 ms apart, letting textual symptoms directly query brief acoustic events like crackles and wheeze onsets.

4. Thinker-A²CA active curriculum agent. An LLM-based central controller that tracks the Diagnoser's error profiles and calibrated confidence, then schedules targeted synthesis of hard-to-diagnose samples, turning data augmentation from a passive remedy into a self-correcting training curriculum.

Main Findings

• ICBHI state of the art. On the standard ICBHI 4-class benchmark with the official 60-40% split, Resp-Agent reaches a score of 72.70 (sensitivity 79.29%, specificity 66.10%), beating the previous best (67.55) by more than 5 absolute points, despite using an LLM+Longformer backbone rather than heavily pretrained audio transformers.

• Large gains on the cross-domain benchmark. On Resp-229k's held-out test set, the full Diagnoser reaches 0.8494 accuracy and 0.2118 macro-F1 without synthetic augmentation, rising to 0.8870 accuracy and 0.5980 macro-F1 in the generator-balanced regime, versus 0.720/0.782 accuracy and 0.1935/0.5360 macro-F1 for an audio-only Conformer baseline.

• The planner matters more than the budget. At a matched budget of 50k synthetic clips, random sampling yields 0.442 macro-F1, class-prior rebalancing 0.512, static uncertainty sampling 0.546, and Thinker-A²CA 0.598 (tail-class macro-F1 0.421). The agent is roughly twice as sample-efficient, capturing about 52% of its total gain with only 10k clips.

• No single heuristic explains the planner's advantage. Factorized variants using only rare labels, only hard cases, or only hard domains each reach 0.489-0.512 macro-F1 at 30k clips, while the multi-factor Thinker reaches 0.541. The iterative, combined strategy cannot be reduced to a handcrafted rule.

• Synthetic data beats loss re-weighting. Class-weighted cross-entropy and focal loss improve macro-F1 only from 0.212 to 0.248 and 0.267 respectively; the Thinker-guided generation path adds substantially more.

• Robust across source domains. In leave-one-source-out evaluation, Thinker-A²CA (0.532/0.383 macro-F1/tail) consistently outperforms class-prior (0.473/0.334) and no-synthesis (0.237/0.086) across all held-out sources.

• Content and style are disentangled. Style-swap tests (fixing a rare pathology, varying the reference timbre) give 0.91 style similarity and 97.9% pathology accuracy; content-swap tests give 0.93 style similarity and 96.1% pathology accuracy, confirming the generator controls "what" and "how it sounds" independently.

• Architecture components are all necessary. Ablations show late fusion with LLM-rendered EHR reaches 0.790 accuracy / 0.160 macro-F1; modality weaving without anchors destabilizes (0.650 accuracy); adding strategic anchors restores stability and lifts performance to 0.849 / 0.212.

Methodology in Plain English

The authors build three pieces that talk to each other.

First, they assemble a large dataset by aggregating five existing public respiratory sound databases, cleaning out corrupted clips, and running a small LLM over the heterogeneous metadata to produce consistent clinical summaries for each recording. A second-stage audit with rule checks, a stronger model as verifier, and human spot-review guards against hallucinated narratives.

Second, they build a sound generator in two stages. A small text LLM is taught to predict discrete audio tokens, guided by two inputs: a text description of the disease to produce, and a style embedding extracted from a reference recording that dictates timbre and device character. Because the model cannot peek at the answer, a masking trick randomly hides about 10% of preceding tokens during training. A separate diffusion-style decoder then turns those discrete tokens back into a full waveform, using flow matching, which is better at preserving phase and sharp transient events than spectrogram-based reconstruction.

Third, a diagnostic model reads the clinical text and the audio together as one interleaved sequence. Rather than full quadratic attention across all tokens, it uses a sliding window plus a sparse set of global tokens: the classifier token, a description sentinel, and audio anchors sampled every fourth frame. This keeps computation linear in sequence length while guaranteeing that a text phrase like "wheeze" can attend directly to acoustic events roughly 80 ms apart.

Coordinating all of this is an LLM-based planner. It examines where the Diagnoser is failing, decides which disease classes and source domains need more training examples, and issues synthesis requests to the generator. The loop repeats, so the system continuously patches its own weak spots rather than augmenting uniformly.

Why This Matters

The paper shows that coupling generation and diagnosis inside a single feedback loop substantially outperforms treating augmentation as a fixed, one-shot preprocessing step. This has broad implications for any data-scarce medical domain, not just respiratory audio: the general principle is to let the model's own failure modes dictate what synthetic data gets created.

Real-world applications:

• Clinical decision support at the point of care, where a system listens to a recording and surfaces audio-informed diagnostic suggestions alongside patient history.
• Medical education, where trainee clinicians can study rare pathological sounds that are difficult to encounter naturally or that no single teaching hospital has in sufficient volume.
• Augmentation for underdiagnosed conditions such as Kawasaki disease, pulmonary hemosiderosis, or lower respiratory tract infections, where existing datasets contain only a handful of examples.
• Remote and telehealth respiratory screening, where the strict cross-device and cross-institution evaluation protocol mirrors the messy reality of consumer microphones and different stethoscope models.

Industry relevance: Companies developing digital stethoscopes, remote patient monitoring platforms, and wearable pulmonary sensors face exactly the data scarcity and device-shift problems this paper targets. The benchmark's source-disjoint protocol is directly relevant to regulatory and validation conversations, since it demonstrates performance on devices and recording setups never seen during training. The open-sourced dataset, models, and code lower the barrier for commercial entrants.

Future Directions

• Clinical validation with real patients. The system is explicitly not a certified medical device. Prospective studies comparing its outputs against pulmonologist consensus are the obvious next step before any deployment claim.
• Clinician-in-the-loop synthesis review. The authors envision deployments where specialists inspect and correct generated edge cases, but the paper does not yet implement or evaluate this interaction. How much human oversight is needed per synthetic sample is an open question.
• Extending the agent loop to other modalities. The same closed-loop architecture could plausibly incorporate imaging, spirometry, or lab results, but whether modality weaving and anchor-based attention generalize to those inputs is untested.
• Scaling and biological fidelity of synthesis. The evaluation measures acoustic similarity and downstream diagnostic utility, not whether generated sounds would be judged clinically plausible by trained ears. Whether the generator can produce genuinely novel pathological presentations rather than interpolations of existing data remains unresolved.
• Governance of synthetic medical data. Even though the source corpora are de-identified, releasing synthetic audio derived from patient recordings raises unresolved questions about consent, provenance, and misuse that the ethics statement acknowledges but does not fully resolve.

Target Audience

This paper is most valuable to machine learning researchers working on medical audio, multimodal fusion, or generative data augmentation, particularly those dealing with long-tailed class distributions and cross-domain generalization. It also speaks to clinical informatics groups evaluating automated auscultation, biomedical engineers building digital stethoscope systems, and dataset curators interested in the pipeline for converting heterogeneous metadata into standardized clinical narratives. Readers without a background in transformer architectures or generative modeling will find the methods dense, though the high-level framing of the generate-diagnose loop is broadly accessible.

Authors’ abstract

Deep learning-based respiratory auscultation is currently hindered by two fundamental challenges: (i) inherent information loss, as converting signals into spectrograms discards transient acoustic events and clinical context; (ii) limited data availability, exacerbated by severe class imbalance. To bridge these gaps, we present Resp-Agent, an autonomous multimodal system orchestrated by a novel Active Adversarial Curriculum Agent (Thinker-A$^2$CA). Unlike static pipelines, Thinker-A$^2$CA serves as a central controller that actively identifies diagnostic weaknesses and schedules targeted synthesis in a closed loop. To address the representation gap, we introduce a Modality-Weaving Diagnoser that weaves EHR data with audio tokens via Strategic Global Attention and sparse audio anchors, capturing both long-range clinical context and millisecond-level transients. To address the data gap, we design a Flow Matching Generator that adapts a text-only Large Language Model (LLM) via modality injection, decoupling pathological content from acoustic style to synthesize hard-to-diagnose samples. As a foundation for these efforts, we introduce Resp-229k, a benchmark corpus of 229k recordings paired with LLM-distilled clinical narratives. Extensive experiments demonstrate that Resp-Agent consistently outperforms prior approaches across diverse evaluation settings, improving diagnostic robustness under data scarcity and long-tailed class imbalance. Our code and data are available at https://github.com/zpforlove/Resp-Agent.