The Geometry of Dialogue: Graphing Language Models to Reveal Synergistic Teams for Multi-Agent Collaboration

AI summary

Overview

Research area: Multi-agent large language model (LLM) systems, automated team composition, and graph-based analysis of model interactions.

Technical level: Intermediate. The paper is readable for someone familiar with LLMs and basic graph concepts (nodes, edges, community detection), but the experimental setup involving ten locally served models and multiple benchmarks assumes some background.

Scope: The paper proposes and tests an interaction-centric framework that builds a "language model graph" from pairwise model conversations and uses community detection to discover synergistic LLM teams, without any prior knowledge of model internals, training data, or task performance.

What This Paper Is About

Multi-agent LLM systems can outperform single models, but only if the right models are teamed together, and picking that team is hard because most models are opaque — many are API-only, and even open models rarely release full training data. Existing automated team-composition work is task-driven and top-down: it decomposes a task into roles and then recruits agents, which downplays the latent relationships between models. This paper asks whether the intrinsic structure of model-to-model interaction alone can reveal which models work well together, and whether the resulting teams actually perform better on downstream benchmarks.

Key Contributions

1. An automatic LLM team composition framework based on inter-model conversations that requires no prior knowledge of internal architecture, training data, or external test data.
2. Empirical demonstration that the graph-based method identifies clusters of models consistent with their latent characteristics — grouping math-specialized models with math-specialized models, medical with medical, and isolating a small-scale model on its own.
3. Performance validation on downstream benchmarks showing that teams built from the detected clusters beat randomly assembled teams and approach the performance of manually curated teams grouped by known specialization.
4. A reframing of model selection as a "geometry of dialogue" problem, where the semantic coherence of utterances in embedding space serves as a proxy for conversational quality and functional similarity between models.

Main Findings

• Conversation geometry tracks model similarity. The relationship value between two models is the summed cosine similarity of utterance embeddings across complete turns. The authors hypothesize that similar models trace a dense, close trajectory in embedding space, while dissimilar models diverge — and the clustering results are consistent with this.

• Topic priming is decisive for cluster quality. Under the (a) general prompt, communities 1 and 2 were heterogeneous mixtures of general-purpose, mathematical, and medical models. Under the (b) mathematical prompt, community 2 became a coherent math group containing Mathstral-7B-v0.1, Qwen2-Math-7B-Instruct, Qwen2-7B-Instruct, and Qwen2-0.5B-Instruct. Under the (c) medical prompt, community 2 grouped two of the three medical-specialized models with a general-purpose model.

• Small or diverging models are isolated. gemma-3-1b-it was separated into its own community (community 3) in the general, mathematical, and medical conditions.

• Conversation content corroborates the graph. In the mathematical scenario, dialogue within community 2 stayed focused on a specific mathematical problem, while dialogue between community 2 and community 3 quickly shifted to a high-level philosophical discussion. In the medical scenario, intra-community dialogues were clinically focused while inter-community dialogues discussed artificial intelligence in medicine generally.

• Detected teams approach curated teams on math. Under the mathematical prompt, community 2 scored 44.9 on MMLU, 71.2 on GSM8K, 88.6 on MATH, 60.0 on MedQA, 53.8 on MedMCQA, giving a Math-Avg of 56.7 versus 56.9 for the math type-based baseline. Its overall average was 60.2.

• Detected teams approach curated teams on medicine. Under the medical prompt, community 2 scored 39.6 on MMLU, 77.0 on GSM8K, 63.9 on MATH, 39.8 on MedQA, 70.7 on MedMCQA, giving a Med-Avg of 73.8 versus 76.4 for the medical type-based baseline. Its overall average was 59.1.

• General-topic teams still beat random. Even without a domain signal, community 2 under the general prompt reached an overall average of 61.0, comparable to the single-model baseline (61.6) and the all-models baseline (60.9), and well above Random@3models (54.4). Community 3 under the general prompt performed poorly at 37.2 overall.

• Naive ensembles underperform. All-models achieved 60.9 overall and Random@3models achieved 54.4, both below the topic-primed community teams, supporting the need for selective team composition.

Methodology in Plain English

The method has three phases.

Phase 1 — Conversation generation. Take N models and generate one conversation for every unique pair, giving all O(N²) pairs. All models share the same system prompt, which instructs them to continue a debate with concise, negative or critical, logically sound responses, and to say "END DISCUSSION" if the discussion is exhausted or non-contributive. One model is randomly chosen to speak first, starting from a fixed starter prompt. The models then alternate for at most K_max = 5 turns, or until either model emits the termination token.

Phase 2 — Graph construction. Each utterance is embedded with a pre-trained sentence embedding function. For each pair, the relationship value is the sum over complete turns of the cosine similarity between the two utterances in that turn. Only complete turns (pairs of consecutive utterances) are counted. Pairs whose relationship value meets or exceeds a threshold τ (set to the median of all relationship values) become edges in an undirected, weighted graph whose nodes are the models. The threshold sparsifies the graph and filters weak or noisy interactions. Relationship values were averaged over five conversation runs per pair to reduce sequential bias.

Phase 3 — Community detection. The Louvain method, as implemented in NetworkX, is applied to the graph to extract densely connected communities, which are treated as candidate collaborative teams.

Experimental setup. Ten models across four types: (i) relatively small-scale general-purpose (Mistral-7B-Instruct-v0.2, Meta-Llama-3-8B-Instruct, Qwen2-7B-Instruct); (ii) small-scale general-purpose (Qwen2-0.5B-Instruct, gemma-3-1b-it); (iii) mathematics fine-tuned (Mathstral-7B-v0.1, Qwen2-Math-7B-Instruct); (iv) medical fine-tuned (BioMistral-7B, Bio-Medical-Llama-3-8B, medgemma-4b-it). Models were served with vLLM and conversations managed with AutoGen. Embeddings came from multilingual-e5-large. Three starting prompts were used: a general open-ended prompt, a mathematics-focused prompt, and a medical-focused prompt.

Evaluation. Zero-shot accuracy on MMLU subsets for math (abstract_algebra, college_mathematics, college_physics) and medicine (clinical_knowledge, college_biology, college_medicine), plus GSM8K and MATH-500 for math, and MedQA and MedMCQA for medicine. Reasoning-intensive datasets used a Chain-of-Thought prompt, except for BioMistral-7B and Bio-Medical-Llama-3-8B, where CoT caused a collapse in responses. Team answers came from majority vote across member models, averaged over five independent trials. Baselines were single-model (best model, Qwen2-7B-Instruct, with self-consistency over ten responses), all-models (all ten voting), Random@3models (average over five random three-model teams), and type-based (groups formed by known specialization).

Why This Matters

Impact on research. The work moves automated team composition away from task-first, top-down agent recruitment toward a bottom-up, interaction-first view. It offers evidence that a model's latent functional specialization can be read off from how it converses with peers — a signal available even when weights, training data, and benchmark results are not. It also connects multi-agent LLM design to established social-graph community detection.

Real-world applications:

• Enterprise model routing, where an organization has many API-accessible models and needs to pick a collaborating subset without access to internals.
• Domain-specific assistant assembly, for example composing a reliable math or clinical reasoning team from a mixed model pool.
• Model portfolio management, where a vendor or platform wants to catalogue which of its hosted models naturally cluster together before offering bundled multi-agent products.
• Screening out weak participants, since the method isolates models that degrade collective performance into their own sparse communities.

Industry relevance. The framework is directly deployable by teams already serving multiple models behind APIs, since it needs only their outputs. The paper is authored by researchers in Hitachi's Research and Development Group and was accepted at the AAAI-26 Workshop on LaMAS 2026 (Oral), indicating industrial interest in automated multi-agent orchestration.

Future Directions

1. Sensitivity analysis. The outcomes depend on the embedding model, the threshold τ, and the community detection algorithm; the authors state that a comprehensive sensitivity analysis is required to understand their impact and give guidance for other contexts.
2. Better definitions of a "good" conversation. Cumulative cosine similarity could reward conversations where models simply agree. Measures that capture constructive progress in embedded space are needed to identify more synergistic teams.
3. Scalability. Pairwise conversation generation scales as O(N²), a bottleneck for hundreds or thousands of models. The authors report preliminary explorations with an NN-Descent-like approximate nearest neighbor approach that both reduced complexity and yielded clustering similar to topic priming.
4. Richer collaboration protocols and hybrid team design. Evaluation used simple majority vote; testing detected communities under multi-round debate is left open. The authors also suggest combining communities (for example, a math community with a medical community) and integrating interaction-centric discovery with existing task-centric frameworks as a catalogue of promising subteams.

Target Audience

Researchers and engineers working on multi-agent LLM systems, model orchestration, and automated agent selection will get the most from this paper. It also suits practitioners building ensembles from mixed proprietary and open models who cannot inspect model internals, and readers interested in applying graph and community detection methods to language model analysis. Readers without familiarity with embeddings, cosine similarity, and graph clustering will need some background to follow the methodology in detail.

Authors’ abstract

While a multi-agent approach based on large language models (LLMs) represents a promising strategy to surpass the capabilities of single models, its success is critically dependent on synergistic team composition. However, forming optimal teams is a significant challenge, as the inherent opacity of most models obscures the internal characteristics necessary for effective collaboration. In this paper, we propose an interaction-centric framework for automatic team composition that does not require any prior knowledge including their internal architectures, training data, or task performances. Our method constructs a "language model graph" that maps relationships between models from the semantic coherence of pairwise conversations, and then applies community detection to identify synergistic model clusters. Our experiments with diverse LLMs demonstrate that the proposed method discovers functionally coherent groups that reflect their latent specializations. Priming conversations with specific topics identified synergistic teams which outperform random baselines on downstream benchmarks and achieve comparable accuracy to that of manually-curated teams based on known model specializations. Our findings provide a new basis for the automated design of collaborative multi-agent LLM teams.