Visible Reasoning: User-Facing Decision Transparency for Generative AI Systems

AI summary

Overview

Research area: Human–Computer Interaction (cs.HC), specifically user-facing transparency and explainability for tool-augmented generative AI agents.

Technical level: Intermediate. The paper is readable by product designers and HCI researchers, but it assumes familiarity with embeddings, cosine similarity, and agent tool-calling patterns.

Scope: The paper introduces Filtered Decision Logs (FDLs), a user-facing interface artifact that shows which tool outputs most shaped an AI agent's answer, ranked by a deterministic four-signal scoring method, and demonstrates it across three scenarios with a 50-participant formative evaluation.

What This Paper Is About

When a generative AI agent calls external tools (APIs, databases, search engines) and produces a recommendation, users currently see only the final answer. They cannot tell whether that answer was driven by solid fundamentals or by recent news hype, and they have no structured way to correct it. The paper proposes a transparency layer that surfaces a short decision rationale, the supporting data points from tools ordered by influence, and relevance indicators, so users can decide whether to accept the answer, supply missing context, or exclude a source and re-run.

Key Contributions

1. Filtered Decision Logs (FDLs): A user-facing transparency artifact with three components — a concise decision rationale, influence-ordered supporting data points derived from tool outputs, and relevance indicators (High, Medium, Low).

2. Provenance reconstruction: A trace-based method that walks an agent's execution trace (recorded as a "context tree" of parent–child spans) to identify data nodes, with a fallback extraction strategy that pulls tool results from the global conversation messages array when the tree walk finds fewer data nodes than expected.

3. Deterministic influence ranking: A scoring method combining four signals — Final Answer Similarity, Average Relevancy, Persistence, and Structural Proximity (DEPTH) — with adaptive weight redistribution for evidence with no reasoning ancestors, validated by counterfactual tool-removal experiments.

4. Multi-domain prototype and formative evaluation: A working prototype demonstrated across career advising (5 tools), investment analysis (4 tools), and travel planning (4 tools), with feedback from 50 participants.

Main Findings

• High overall approval: 46 of 50 participants (92%) rated the influence-ranked evidence as helpful for understanding and trusting the recommendation.
• Professionals approved unanimously: All 25 working professionals gave a positive rating (100%); 21 of 25 university students did (84%). The four negative student ratings cited difficulty interpreting the signal breakdown (AVG and PERSIST), not objections to influence ranking itself.
• Counterfactual validation succeeded in all three scenarios: Removing the top-ranked source changed the answer more than removing the bottom-ranked source, measured as embedding similarity to the original answer. Career: 0.72 (remove top) versus 0.91 (remove bottom); Investment: 0.78 versus 0.93; Travel: 0.75 versus 0.89.
• Career scenario top driver: social-media-profile scored 0.74 (High), above project-portfolio 0.66, academic-records 0.60, personality-assessment 0.56 (all Medium), and job-market-data 0.37 (Low). Excluding social-media-profile and re-running shifted the recommendation from "Policy Analyst" to "Data Scientist," with project-portfolio rising to the top position and academic-records rising from 0.60.
• Investment scenario top driver: company-financials scored 0.70 (High), followed by economic-indicators 0.68, stock-price-detail 0.67, and financial-news 0.66 — indicating the agent weighted fundamentals and macro context over news sentiment, despite a designed tension of bullish news against a 68× trailing P/E.
• Travel scenario top driver: destination-weather scored 0.68 (Medium), the lowest top-ranked score of the three scenarios, in a scenario built around a tension between cherry-blossom season and budget optimization.
• Interaction patterns observed: Participants consistently began by examining the top-ranked source; several spontaneously excluded the top source to "see what happens without it" without instruction; professionals who expanded the signal breakdown reported higher confidence, particularly when Final Answer Similarity was high.
• Scores are explicitly not additive or causal: The paper stresses that a score of 0.70 means "high semantic alignment with the answer," not "this source contributed 70% of the answer," and that scores do not sum to 1.0 across items.

Methodology in Plain English

The design separates two pipelines that never interact, which the paper calls "evaluation independence." The first is an ordinary tool-using agent built on Amazon Bedrock with Claude 3 Sonnet using the ReAct pattern; it reasons about which tools to call, observes results, and produces a final answer while its execution trace is recorded as a tree of stages.

The second is a 7-stage linear pipeline that uses no generative AI at all — only embeddings from Amazon Titan Embed Text v2 and deterministic arithmetic. It walks the execution tree to find tool-result nodes, builds parent pointers, and collects the chain of LLM reasoning steps above each node. It then embeds the evidence, the final answer, and the reasoning steps, and computes four similarity-based signals:

• Final Answer Similarity (FA): how closely the tool output matches the final answer.
• Average Relevancy (AVG): average similarity between the tool output and each reasoning step.
• Persistence (PERSIST): the fraction of reasoning steps whose similarity exceeds a threshold (default 0.30), capturing breadth rather than intensity of reference.
• Structural Proximity (DEPTH): 1/(1+n), where n is the number of LLM reasoning ancestors — a direct tool result with no intermediaries scores 1.0.

These are combined with default weights α=0.40 (FA), β=0.30 (AVG), γ=0.20 (PERSIST), δ=0.10 (DEPTH), summing to 1.0. Because AVG and PERSIST are structurally zero when a tool result has no LLM ancestors (which would cap scores at roughly 0.50), the pipeline redistributes those weights per evidence item, yielding α′=0.80 and δ′=0.20 while preserving the 4:1 FA-to-DEPTH ratio. The UI flags when adaptation was applied. Items are then sorted, top-K selected, and labeled High (≥0.70), Medium (0.40–0.70), or Low (<0.40).

Evaluation used deterministic mock tools returning structured JSON, each scenario containing a designed tension — conflicting signals that make influence scores meaningful. Validation combined counterfactual tool removal with a guided formative study. Participants inspected rankings, identified which source they believed most influenced the recommendation, and used an exclude/re-run interaction, then gave a single thumbs-up/thumbs-down rating on whether the influence-ranked evidence helped them understand and trust the recommendation, plus optional free-text comments. The prototype is a TypeScript application: a Fastify agent backend with re-ask support via ignoredTools and systemPromptOverride parameters, the FDL pipeline exposed as POST /api/explain, and a React frontend split 60% chat / 40% influence panel. Testing included 75+ tests with property-based tests (fast-check) validating scoring invariants.

Why This Matters

Impact on research: The paper reframes explainability for agents from "explain the model's tokens" to "show which runtime artifacts shaped this specific answer." Its central argument is that chain-of-thought is not a governance mechanism — citing Turpin et al. for the finding that CoT traces do not always faithfully represent the model's actual decision process — and that an external, deterministic measurement is auditable in a way self-narrated reasoning is not. It positions FDLs at the tool level rather than the token or feature level of LIME and SHAP, and at the runtime level rather than the training-data level of influence functions and Data Shapley.

Real-world applications:

• Financial advisory interfaces: surfacing whether a "buy" recommendation came from fundamentals, macro indicators, or news sentiment before a user acts on it.
• Career and educational advising: revealing that a recommendation was driven by social media signals rather than academic record, and letting the user exclude that source and re-run.
• Enterprise agent observability: converting existing OpenTelemetry-style traces into a governance surface for end users rather than only for developers.
• Travel and consumer planning agents: letting users distinguish season-driven from budget-driven itineraries.

Industry relevance: The authors are employed by Amazon Web Services, and the prototype runs on Amazon Bedrock, Claude 3 Sonnet, and Titan Embed Text v2, so the work maps directly onto existing cloud agent stacks. The paper argues that many platforms already collect the observability telemetry needed; the missing piece is the transformation into user-comprehensible ranked evidence. The exclude/re-run interaction also produces a natural feedback signal — which sources users exclude and whether they accept the revised answer — that could later calibrate ranking weights.

Future Directions

1. Controlled summative evaluation: a between-subjects study comparing FDL-augmented versus standard interfaces, measuring trust, understanding, and actionability with statistical significance testing, plus comparison of FDL rankings against human expert rankings.
2. From correlation to causation: the current signals are embedding similarity, which the authors acknowledge is not causal attribution. Candidate directions include causal attribution via counterfactual agent runs, and learning influence weights from user feedback or configuring domain-specific weight profiles from expert input.
3. Ablation and robustness testing: varying the number of signals (FA only, FA+AVG, all four) to measure effects on user trust, replacing deterministic mock tools with live APIs to test score stability, and testing across multiple LLM providers beyond Claude 3 Sonnet.
4. Extending scope beyond single-agent, single-turn settings: how influence should propagate across agent delegation boundaries in multi-agent systems, and how source influence decays or accumulates across turns in multi-turn conversations.

Target Audience

Product designers and HCI researchers building trust interfaces for AI systems; ML engineers and platform teams implementing agent observability and explainability features; AI governance and risk practitioners who need auditable rather than self-reported transparency; and readers interested in the limits of chain-of-thought as an explanation mechanism. The paper is also accessible to graduate students in HCI or applied AI looking for a concrete, implementable starting point for evidence-level attribution in tool-using agents. Readers should note the authors' own framing: the weights are configurable priors, not learned parameters, and the study is formative rather than a controlled experiment.

Authors’ abstract

Across multiple waves of computing, public trust in new technology has followed the emergence of governance mechanisms that made dominant risks legible and manageable. Generative AI shifts this pattern: these systems synthesize across heterogeneous sources, and users increasingly experience them as reasoning systems. The central question changes from "did it perform correctly?" to "can we trust how it arrived at its conclusion?" This paper introduces Filtered Decision Logs (FDLs), a user-facing transparency mechanism that makes decision formation visible without exposing raw chain-of-thought. FDLs pair each response with (i) a concise decision rationale, (ii) influence-ordered supporting data points derived from tool outputs, and (iii) relevance indicators showing which sources most shaped the conclusion. To operationalize evidence selection, we propose a trace-based method that reconstructs observability telemetry into a provenance tree and ranks evidence using four deterministic semantic and structural signals - Final Answer Similarity, Average Relevancy, Persistence, and Structural Proximity - combined via adaptive weighted scoring. We demonstrate FDLs across three domains: career advising (5 tools), investment analysis (4 tools), and travel planning (4 tools). In a formative evaluation with 50 participants (25 students, 25 working professionals in AI development), 92% rated the influence-ranked evidence as helpful for understanding and trusting the system's recommendation. Counterfactual tool-removal experiments confirm that removing the top-ranked source changes the answer more than removing a low-ranked source, validating that the ranking tracks decision dependence. We conclude that evidence-ranked transparency helps users identify misleading signals and missing context, enabling more confident acceptance, correction, or re-run of system decisions.