ConSensus: Multi-Agent Collaboration for Multimodal Sensing

AI summary

Overview

Research area: LLM-based multimodal sensor fusion, multi-agent LLM collaboration, and training-free reasoning over heterogeneous time-series sensor data (physiological and physical activity sensing).

Technical level: Intermediate — readable without deep expertise, but assumes familiarity with large language models, prompting, and basic sensor/benchmark concepts.

Scope: The paper introduces ConSensus, a training-free multi-agent framework that decomposes multimodal sensing into per-modality LLM agents and aggregates their interpretations through a single-round "hybrid fusion" that combines semantic reasoning with statistical consensus.

What This Paper Is About

Multimodal sensing tasks — such as inferring affective state or recognizing activities — require integrating evidence from heterogeneous sensors whose reliability differs from moment to moment. When a single LLM is given all sensor features at once, it tends to reason incompletely, and when the outputs of multiple agents are merged, LLM-based judges over-weight certain "important" modalities while majority voting breaks down when sensors are noisy or missing. ConSensus aims to get the best of both by pairing specialized per-modality agents with a fusion protocol that arbitrates between knowledge-grounded and consensus-grounded reasoning.

Key Contributions

1. A modality-aware multi-agent framework for multimodal sensing. ConSensus decomposes heterogeneous sensor inputs into specialized agents, one per modality, and operates without any additional training, allowing direct deployment across tasks.
2. A hybrid fusion strategy. The framework balances semantic aggregation (cross-modal reasoning by an LLM judge) against statistical consensus (majority voting), addressing the complementary failure modes of knowledge bias and sensor failure.
3. Broad empirical evaluation. ConSensus is tested on five multimodal sensing benchmarks, reporting an average accuracy improvement of 7.1% over the single-agent baseline.
4. Efficiency demonstration. The single-round fusion protocol matches or exceeds state-of-the-art iterative multi-agent debate methods while achieving a 12.7× reduction in average fusion token cost.

Main Findings

• Modality decomposition helps on its own. ConSensus improves average accuracy over the Single-Agent baseline by 7.1%. Even without hybrid fusion, both the semantic-only and statistical-only variants showed substantial gains, indicating that splitting reasoning by modality is itself valuable.
• Self-Consistency is a weak remedy. Self-Consistency yielded only a marginal average gain of 0.6% and remained substantially below ConSensus.
• Semantic and statistical fusion fail in different places. Statistical fusion outperformed semantic fusion by an average margin of 1.5%, but the best choice varied by dataset: semantic fusion led on PAMAP2 by 2.5%, while statistical fusion dominated on WESAD by 10.2%, where the semantic agent over-weighted ECG-derived predictions due to knowledge bias.
• Hybrid fusion resolves the trade-off. The hybrid fusion agent achieved the highest average accuracy and surpassed both single-fusion variants on SleepEDF, ActionSense, and MMFit.
• Competitive with debate at far lower cost. ConSensus reached 72.3% average accuracy versus 72.7% for ReConcile, while debate baselines required 76K additional tokens per sample and fusion agents required 6K. ConSensus reduced aggregation tokens by 12.7× averaged across debate baselines; ReConcile alone needed 78.6K aggregation tokens per inference versus ConSensus's 6K.
• Robustness under missing modalities. When 10%, 30%, and 50% of modalities were randomly omitted, ConSensus maintained an advantage over the Single-Agent baseline, reported as averages of 7.1%, 4.5%, 8.2%, and 1.2% "across the respective missingness ratios," as stated in the paper. Statistical fusion collapsed to 41.4% accuracy at 50% missingness, while semantic fusion maintained 59.9%; ConSensus outperformed statistical fusion by 9.1% at 30% missingness and 18.4% at 50% missingness.
• Performance varies by backbone. With gpt-oss-120B (average: Single-Agent 0.654, ReConcile 0.739, ConSensus 0.713), ReConcile outperformed ConSensus, suggesting larger models benefit more from debate. On Llama models, ConSensus consistently beat ReConcile while using up to 13.1× fewer tokens.
• Largest gains on a small model. On Llama-3.1-8B-Instruct, the Single-Agent baseline scored 29.3%, ReConcile improved it by 8.6%, and ConSensus improved it by 16.3%.

Methodology in Plain English

The researchers start from three observations drawn from illustrative examples on the WESAD dataset using gpt-oss-20B: a single LLM asked to reason over all modalities at once often overlooks some sensor evidence; an LLM "judge" that combines agent outputs tends to lean on prior domain knowledge and over-trusts certain signals (such as ECG); and plain majority voting fails when sensors drop out or are corrupted, because votes stop being reliable and independent.

ConSensus addresses this with four roles. Each modality agent sees only one sensor's features plus the task description and produces a prediction and a written rationale. A semantic fusion agent reads all modality outputs and reasons across them to make a knowledge-grounded prediction. A statistical fusion agent takes the majority-vote result across the modality agents and produces a rationale supporting it. Finally, a hybrid fusion agent acts as coordinator, comparing the semantic and statistical proposals instance by instance and choosing the final prediction.

Inputs are hand-crafted statistical features from each sensor (for example, mean, standard deviation, and spectral measures), presented in text prompts with a one-shot example per class. All inference uses temperature 0 for determinism. The method is compared against a concatenated Single-Agent prompt, Self-Consistency, Self-Refine, and four multi-agent debate frameworks (Debate, MAD, CMD, ReConcile) run over the same modality agents, with debate rounds fixed at two.

Why This Matters

The work shows that careful role decomposition and a structured, single-round fusion step can extract more from heterogeneous sensor streams than either monolithic prompting or expensive iterative debate. It reframes sensor fusion as a reasoning-architecture problem rather than a purely learned-alignment problem, and it does so without training — lowering the barrier to adopting LLMs for sensing.

Real-world applications:

• Context-aware assistance: combining smartphone and smartwatch IMU to infer situations such as driving, being in the office, or staying at home.
• Health monitoring: jointly interpreting ECG and PPG to give early feedback on abnormal conditions.
• Activity and fitness tracking: recognizing gym exercises and daily activities from wearable accelerometers, gyroscopes, and heart-rate sensors.
• Robust wearable deployment: maintaining accuracy when earbud, watch, or chest sensors drop out or degrade, which the paper targets through its missing-modality and noise experiments.

Industry relevance: The 12.7× token reduction and training-free design matter for commercial deployments where inference cost and latency are constraints, and the paper's results on small models (such as Llama-3.1-8B-Instruct) suggest the approach is viable on resource-constrained hardware. The authors also describe ConSensus as deployable on commodity devices like smartphones, smartwatches, and earbuds, and provide task-agnostic prompt configurations in the appendix for practitioners to adapt.

Future Directions

• Scaling the evaluation. The authors note that experiments used feasible subsets of each dataset rather than the full dataset, and call for testing scalability to larger datasets and long-term sensing populations while reducing inference overhead.
• Beyond classification. The framework is currently limited to classification tasks because no established benchmark exists for LLM-based multimodal sensing on broader task types; extending to human-centric reasoning or subjective judgment would require new data collection protocols and human annotations.
• Explicit uncertainty-aware fusion. The paper suggests incorporating reliability signals as fusion weights via confidence-weighted voting, tool-augmented modality agents that estimate signal quality, or historical reliability modeling.
• Combining with other prompting and training techniques. The authors deliberately did not stack Self-Consistency, Self-Refinement, or confidence-based protocols like ReConcile on top of ConSensus, nor did they fine-tune agents or use retrieval-augmented generation — all identified as promising extensions.

Target Audience

This paper suits researchers and engineers working on multimodal sensor fusion, wearable and mobile sensing, and LLM-based agents. It is most useful to practitioners who want a training-free, cost-efficient alternative to monolithic prompting or multi-round debate, and to researchers studying the failure modes of LLM judges versus statistical aggregation. Readers need only a working grasp of LLM prompting and basic machine learning evaluation to follow the core arguments.

Authors’ abstract

Large language models (LLMs) are increasingly grounded in sensor data to perceive and reason about human physiology and the physical world. However, accurately interpreting heterogeneous multimodal sensor data remains a fundamental challenge. We show that a single monolithic LLM often fails to reason coherently across modalities, leading to incomplete interpretations and prior-knowledge bias. We introduce ConSensus, a training-free multi-agent collaboration framework that decomposes multimodal sensing tasks into specialized, modality-aware agents. To aggregate agent-level interpretations, we propose a hybrid fusion mechanism that balances semantic aggregation, which enables cross-modal reasoning and contextual understanding, with statistical consensus, which provides robustness through agreement across modalities. While each approach has complementary failure modes, their combination enables reliable inference under sensor noise and missing data. We evaluate ConSensus on five diverse multimodal sensing benchmarks, demonstrating an average accuracy improvement of 7.1% over the single-agent baseline. Furthermore, ConSensus matches or exceeds the performance of iterative multi-agent debate methods while achieving a 12.7 times reduction in average fusion token cost through a single-round hybrid fusion protocol, yielding a robust and efficient solution for real-world multimodal sensing tasks. The source code is available at https://github.com/nokia/multi-agent-collaboration-for-multimodal-sensing.