Counterfactual-based Agent Influence Ranker for Agentic AI Workflows

AI summary

Overview

Research area: Agentic AI / LLM-based multi-agent systems (the paper uses the term Agentic AI Workflow, AAW), with connections to interpretability, security, and inference-time efficiency.

Technical level: Intermediate. The paper is readable for someone familiar with LLMs and basic machine-learning concepts; the core idea (counterfactual perturbation) is intuitive, while the scoring formulas and evaluation metrics require some background.

Scope: The paper introduces CAIR, a method that ranks the agents inside an agentic AI workflow according to how much each one influences the workflow's final output, using counterfactual changes to agent outputs in an offline phase and a lightweight lookup at inference time.

What This Paper Is About

Agentic AI workflows assemble several LLM-based agents that collaborate toward a shared goal, but there has been no way to measure how much each individual agent actually shapes the workflow's final answer. Existing techniques borrowed from graph theory, network security, or reinforcement learning only perform static structural analysis, which cannot handle workflows that are dynamic and that change which agents are activated depending on the input query. CAIR addresses this gap by performing counterfactual analysis of agent outputs, producing a task- and architecture-agnostic influence ranking that can also be applied at inference time with negligible added latency.

Key Contributions

1. The first agent influence ranker for agentic AI workflows. CAIR is task- and architecture-agnostic and can be used in both offline and online analysis, with negligible added latency to workflow inference time.
2. Enabling LLM-level downstream tasks on multi-agent workflows. CAIR is described as the first method to let downstream tasks designed for LLMs be applied to multi-agent AAWs efficiently during inference.
3. A pioneering interpretability analysis of AAWs, providing understanding of how individual agents relate to the final output.
4. Open resources for the community: the CAIR code, the AAW-Zoo dataset, and the AAW-Zoo-Generator used to create that dataset are released for further research into AAWs across quality and security aspects.

Main Findings

• CAIR outperforms graph-based baselines overall: Using the classical feature importance (CFI) method as a ground-truth proxy, CAIR achieved an overall TRS of 29.27 (23.04), P@3 of 62.6 (18.8), P@2 of 80.95 (12.6), P@1 of 76.1 (9.7) and 1-SFD of 62.1 (12.4). Betweenness centrality (BTW) scored 6.0 (4.2), 33.6 (8.1), 14.0 (4.89), 43.0 (4.8) and 50.2 (5.3); Eigenvector centrality (EV) scored 8.7 (4.9), 36.3 (8.6), 19.0 (4.95), 49.3 (4.1) and 41.9 (8.6); random choice scored 0.3 (0.2), 5.1 (1.5), 6.4 (1.1), 15.9 (1.3) and 34.3 (0.6).
• Sequential architectures are the exception: In sequential AAWs, which have less architecture-related autonomy, EV had higher TRS (26.0 vs. CAIR's 24.0), P@1 (48.0 vs. 35.0) and 1-SFD (46.7 vs. 41.1), while CAIR remained competitive. The paper attributes this to the strong architectural effect in sequential flows, where agents activated late in the workflow have higher influence.
• Clear dominance in orchestrator and router architectures: For orchestrator AAWs, CAIR reached P@1 of 100.0 (0.0), 1-SFD of 65.4 (10.6) and P@2 of 65.5 (15.5), versus BTW and EV at 0.0 (0.0) TRS and 33.0 (16.7) P@3. For router AAWs, CAIR reached TRS 40.0 (38.9), P@3 63.3 (31.4), P@2 93.3 (13.3), P@1 93.3 (20.0) and 1-SFD 79.7 (15.1), while BTW and EV both scored 0.0 on TRS, P@3, P@2 and P@1.
• Online pairing works: In the gift recommender sequential use case, input queries were correctly paired to representative queries of the same functionality based on SBERT cosine similarity, indicating CAIR can be used in online settings.
• Guardrail latency reduced with smaller effectiveness loss: Applying toxicity guardrails only to CAIR's top-ranked half of agents reduced latency by an average of 27.72%, with a 4.76% drop in effectiveness. CFI produced a similar latency gain (26.49%) but a much larger effectiveness drop (11.12%). Per architecture, CAIR's latency improvement (POI) and effectiveness change (EC) were 25.49% / 1.13% for sequential, 17.92% / -13.20% for orchestrator, and 39.75% / -2.22% for router.
• Ablation and stability: Rankings with the highest compatibility to CFI and the lowest variability came from combining all CAIR components (Orig. OC). Deviating from the paper's chosen α = 0.6 and β = 0.4 still yielded similarly good rankings, but pushing α and β toward the extremes of their range degraded 1-SFD. Ranking quality stayed consistent across different representative-query set sizes.
• Human verification favored CAIR: Where CAIR and CFI rankings differed, human verifiers chose the ranking that best matched perceived agent importance, and in most cases this was CAIR's ranking.
• Production-ready hierarchical workflow: On a publicly available LangGraph hierarchical tutorial setup with three supervisor agents and five worker agents, evaluated on three representative queries, CAIR was aligned with the expected rankings while other methods were not.

Methodology in Plain English

CAIR borrows the logic of feature-importance methods from classical machine learning — specifically LIME — and maps it onto a workflow. The workflow is treated like a model, the agents are treated like features, and each agent's input and output are treated like feature values.

The method runs in two phases:

1. Offline phase. For each representative query, CAIR first records the normal activation flow, including every agent's input and output. It then uses an LLM to replace each activated agent's output with a deliberately different but still valid output, and re-runs the remainder of the workflow. It measures how much the final workflow output changed (FOC, calculated as a cosine distance between SBERT embeddings of the original and perturbed final output), how much the agent's own output changed (AOC, to control for perturbation intensity), and how much the activation flow itself changed (WC, computed as an edit distance over the agent sequence). The final influence score is a weighted sum, α·OC + β·WC, where OC = FOC − (AF × AOC) and AF (the amplification factor) is the number of agents remaining to be activated divided by the total number of activated agents. When an agent is activated more than once, its maximum score is taken.
2. Online phase. Each representative query is stored with its embedding and its resulting agent rankings. For a new query, CAIR embeds the query, retrieves the most similar representative query by cosine similarity, and reuses that representative query's rankings. The added computation — one embedding plus similarity comparisons — is negligible compared to a single LLM call.

Why This Matters

Impact on research. The paper argues there are currently no methods for interpreting an agentic workflow's final output in terms of the individual agents involved, and no methods for assessing their influence. It shows that importing static graph-theory or network-security measures is not sufficient because those approaches ignore agent behavior and cannot handle flexible workflows whose activation order is not fixed. CAIR offers a behavioral, inference-time-compatible alternative, along with a generated dataset (AAW-Zoo: 30 use cases, 230 functionalities) and a generator to support further study.

Real-world applications:

• Selective toxicity guardrails: instead of running guardrails on every LLM call (which the paper states can potentially triple inference time), apply them only to the most influential agents.
• Observability and monitoring of multi-agent systems, so operators can see which agents drive outcomes.
• Failure detection mechanisms that focus attention on the agents whose output changes propagate furthest.
• Workplace productivity and coding assistants, the domains the paper cites as current applications of LLM-based agents.

Industry relevance. The paper notes the market for AI agents and AAWs is projected to experience an eightfold increase in adoption over the next six years. It is authored by Fujitsu Research of Europe, and the released code and dataset are aimed at helping organizations that deploy agentic workflows reduce inference latency while keeping safety measures effective.

Future Directions

• Evaluating CAIR on hybrid AAW architectures, such as workflows with several orchestrators and user profiles.
• Testing AAWs with a higher level of agency, where agents are more autonomous.
• Testing with lower access to agent outputs, meaning cases where only the input query and the final workflow output are observable — the paper lists this as a limitation since CAIR currently requires access to each agent's output.
• Extending CAIR to perform downstream tasks by design, for example risk assessment built from CAIR components related to agent failures.

Target Audience

Researchers and practitioners working on multi-agent LLM systems, agentic workflow interpretability, and AI safety infrastructure will benefit most. It is also relevant to engineers building guardrails or observability tooling who need to cut inference latency, and to teams interested in reproducible multi-agent benchmarks, given the released AAW-Zoo dataset and AAW-Zoo-Generator.

Authors’ abstract

An Agentic AI Workflow (AAW), also known as an LLM-based multi-agent system, is an autonomous system that assembles several LLM-based agents to work collaboratively towards a shared goal. The high autonomy, widespread adoption, and growing interest in such AAWs highlight the need for a deeper understanding of their operations, from both quality and security aspects. To this day, there are no existing methods to assess the influence of each agent on the AAW's final output. Adopting techniques from related fields is not feasible since existing methods perform only static structural analysis, which is unsuitable for inference time execution. We present Counterfactual-based Agent Influence Ranker (CAIR) - the first method for assessing the influence level of each agent on the AAW's output and determining which agents are the most influential. By performing counterfactual analysis, CAIR provides a task-agnostic analysis that can be used both offline and at inference time. We evaluate CAIR using an AAWs dataset of our creation, containing 30 different use cases with 230 different functionalities. Our evaluation showed that CAIR produces consistent rankings, outperforms baseline methods, and can easily enhance the effectiveness and relevancy of downstream tasks.