Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries

AI summary

Overview

Research area: Natural language processing applied to research impact assessment, specifically agentic large language model (LLM) systems embedded in clinical and translational science administration workflows.

Technical level: Intermediate. The paper assumes familiarity with LLM agents, tool use, and retrieval, but its central contribution is an evaluation rather than a novel architecture. Readers need no specialized machine learning background to follow the results.

Scope: This paper reports a real-world, human-in-the-loop evaluation of an AI agent that assembles evidence dossiers and drafts one-sentence Translational Science Benefits Model (TSBM) impact summaries for ten career-development scholars at a single Clinical and Translational Science Award (CTSA) hub.

What This Paper Is About

CTSA programs must document the real-world impact of the scholars they fund, but building each scholar's impact record by hand takes staff an estimated 15 hours and does not scale to a full cohort. Much of that impact, such as clinical program leadership, community engagement, cost and adoption data, and policy uptake, leaves no indexed trail and sits outside the databases existing tools query. The authors built a human-in-the-loop AI agent that searches scholarly databases and the open web, assembles a sourced evidence dossier per scholar, and drafts one-sentence TSBM impact summaries for staff review, then evaluated it inside the real reporting workflow rather than in a controlled benchmark.

Key Contributions

1. A deployed evidence-gathering and drafting agent. The system starts from a minimal scholar seed (name, institution, existing grant records), queries OpenAlex, PubMed, ORCID, ClinicalTrials.gov, NIH RePORTER, and the open web, and produces both a structured dossier and a one-sentence benefit statement for each impact finding. It is tuned for recall so reviewers prune rather than fill gaps.

2. An in-workflow evaluation with independent double coding. Two staff members who own the hub's impact-reporting workflow independently reviewed all 507 agent-generated findings using a structured workbook, producing a conservative unanimous usable rate rather than a single-reviewer judgment.

3. A separate profile discovery study. The agent's retrieval recall was compared against human-only search and against a human using a general-purpose AI chat assistant with web search, isolating discovery performance from drafting and review performance.

4. Full release of code, prompts, and configuration. The implementation is published at a public repository, with the caveat that the hub's internal grant records remain private and are not reproducible by outside parties.

Main Findings

• Unanimous usable rate of 81.7%. Both reviewers accepted or edited 414 of 507 findings. Neither reviewer rejected the other 93. Roughly 75% of all findings were accepted unchanged by both reviewers, and each reviewer edited fewer than 5%.

• Review replaced an estimated 15 hours of manual work with a median of 14 minutes per scholar. Individual reviewer times ranged from 8.5 to 25 minutes, making the workload schedulable against a reporting deadline.

• Usable rates varied sharply by dossier section. Career trajectory scored highest at 96.7%, followed by profiles (86.0%) and grants (86.0%), media (80.0%), clinical impact (74.0%), economic impact (70.0%), policy impact (65.0%), and community impact (64.9%). Community and policy findings often fell below the reviewer's threshold for a completed benefit rather than failing on sourcing.

• Sourcing failures dominated the 93 not-usable findings. A weak or non-authoritative source was the largest cause (33 findings), followed by findings that fell short of a completed impact (22) and findings where the scholar's role was too peripheral (16). Eight had no source at all, eight were miscategorized, four had broken or mismatched links, and two were entirely irrelevant. The four broken links were the closest the system came to hallucination.

• About a third of reviewed findings were non-scholarly impact evidence. The four TSBM impact sections held 174 of 507 findings, drawn from institutional pages, news coverage, and the open web alongside journals and registries. Clinical impact drew mostly on journals and trial registries; community and policy impact drew more on institutional pages and media.

• Inter-rater agreement was moderate. Cohen's kappa was 0.43 on the usable-versus-reject decision, 0.35 on the three-way accept/edit/reject scheme, and 0.40 on accept-versus-any-change. The residual disagreement traced to subjective impact thresholds and grant attribution questions. Individually, one reviewer rated 89.9% of findings usable and the other 85.6%.

• Reviewer quality ratings were high. Synthesis accuracy averaged 4.5 of 5 and usefulness 4.8 of 5. Both reviewers said they would use all ten dossiers they rated, and neither added any findings the agent had missed.

• Profile recall was close to human search. On the human-only subset, both the human and the agent scored 0.82 recall across 33 profiles. On the AI-guided subset, the human scored 0.85 versus the agent's 0.76 across 46 profiles; the gap came mostly from Google Scholar profiles, which the agent cannot query.

• Summaries were short and consistently formatted. One sentence averaging 25.4 words (SD 6.2), with accepted examples reporting specific quantitative benefits such as facility registration rates rising from 40% to 89%, colorectal cancer screening rising from 13% to 20%, and annual per-patient costs falling from $64,634 to $16,281.

• Most findings traced to authoritative sources. 84% of all findings and 75% of impact findings cited a funder record, a DOI or PubMed record, a trial registry, or the scholar's own institutional page.

• Resource use per scholar was substantial but bounded. A median of roughly 1.5 million input and 119,000 output tokens, 9 to 25 distinct search queries, and 413 tool calls in aggregate across the cohort.

Methodology in Plain English

The agent works in repeated research runs, up to three per scholar. Each run has three stages. In the gather stage, the agent plans a search objective and executes it, alternating up to 25 times, until it files a coverage report marking every dossier section as found, confirmed absent, or searched and empty. In the assemble stage, it writes retrieved evidence into dossier sections, producing the four impact sections last so each impact can cite evidence already recorded, then drafts one sentence per impact finding. In the critique stage, an isolated pass reads the dossier and the run's action log and flags gaps, duplicates, contradictions, and weak sources for repair in the next run. A final register-revise stage measures each draft against a linguistic signature derived from 76 human-written benefit statements and revises drafts that drift from that style.

The system splits work across two models. A large model plans searches, reasons over evidence, and drafts summaries. A small model handles the high-volume reading: distilling retrieved web pages to requested facts and trimming publication abstracts to their outcomes. Only trimmed content enters the large model's context, which also limits context poisoning from adversarial page text. Every stage that reads untrusted content fences it as data, following the spotlighting family of defenses.

Evidence standards are defined per finding type. Grants prefer the hub's own record or the NIH RePORTER project page; publication-grounded findings prefer the paper's DOI; trials prefer the ClinicalTrials.gov record; profiles and media mentions prefer the scholar's own page or the outlet's own coverage.

The evaluation had two parts at the University of Illinois Chicago's CTSA hub. The profile discovery study split the ten scholars into two non-overlapping five-scholar subsets, searching each with the agent and with one human arm, either human-only web and database search or AI-guided search using a general-purpose chat assistant with verification. Findings were pooled and human-verified, so recall is relative to that pool. The review study had two evaluation staff independently review every finding using a structured workbook that showed the finding, its source, and the agent's reasoning, with publications excluded because they come straight from indexed bibliographic databases. Each finding received an action (accept, edit, or reject) plus a reason tag, and the primary measure was the unanimous usable rate, meaning the share both reviewers accepted or edited. This is a conservative floor by construction. The analysis was descriptive, with no human-first-draft comparison arm.

Why This Matters

Impact on research: In-workflow evaluations of AI systems are rare. The paper cites an estimate that real-world impact evaluations make up about 0.1% of computational linguistics papers, and a clinical workflow review that found only 4 of 288 screened records to be original studies of a live deployment. This study measures the outcomes a reporting workflow actually cares about, namely review time and the action each finding required, rather than benchmark accuracy alone. It also supplies a measurement for the review stage that research-impact assessment previously lacked, borrowing a pattern already established in live systematic reviews, clinical note editing, and enterprise code assistants. Prior work on automated impact assessment drew only on closed sources and missed non-scholarly translational impact entirely; hands-on TSBM profile co-creation reaches that evidence but costs roughly 9 hours per profile with poor investigator response rates.

Real-world applications:

• CTSA hubs and institutional career-development programs assembling scholar impact records for NIH and NCATS reporting.
• Research information systems and current research information system (CRIS) vendors extending beyond publication and grant ingestion into open-web, community, and policy evidence.
• University research offices preparing impact narratives for funders, promotion packets, or policy briefs.
• Any setting where expert review of AI-drafted records is the bottleneck, including systematic review screening and clinical documentation.

Industry relevance: The architecture demonstrates a practical product pattern for high-stakes drafting: recall-oriented generation, per-finding provenance with a preferred authoritative source, an adversarial critique loop, style control through a reference corpus, containment-based prompt injection defense, and human review of every output before it enters a system of record. The moderate inter-rater agreement also signals a real design constraint for anyone deploying similar tools, namely that human judgment thresholds, not model outputs, become the binding source of variance once drafting is automated.

Future Directions

• Multi-site evaluation. The sample is ten scholars at one hub. The pipeline is not site-specific, since only the internal grant seed is local, but replication at other hubs is untested.

• Wiring in economic and policy data sources. Coverage was thinnest in those two domains. Adding United States Patent and Trademark Office patent records, policy-citation indexes such as Overton, legislative databases, and cost or adoption data would extend retrieval the same way trial and publication interfaces already do.

• Moving sourcing fixes upstream. A retrospective rule-based filter flags 21 of 507 findings, including every rejected finding with an empty source field. A denylist of known non-authoritative domains would also screen out the provider-directory pages reviewers repeatedly rejected. Both are planned to run before drafts reach review.

• Measuring what the agent misses. The review study judges only what the agent proposed, and the profile discovery study covers profiles alone, so the completeness of the impact evidence is not established. Separating retrieval gaps from genuine scarcity in economic and policy impact remains an open question.

• Lowering the cost of keeping records current and reducing reviewer threshold ambiguity. Efficiency optimization and clearer guidance on the impact threshold are both open, as is testing model families beyond the OpenAI models used here.

Target Audience

CTSA and CTSI evaluation and impact-reporting staff will find the workflow measurements and reviewer agreement data most directly useful. Researchers in human-AI collaboration and human-computer interaction will value the deliberate treatment of review as a designed part of the contribution. Clinical and translational informatics teams and NLP practitioners building agentic systems for real deployments will benefit from the architecture details and the honest treatment of sourcing failures. Research information system vendors and program officers weighing adoption decisions should read the limitations section alongside the results, particularly the reliance on a staff time estimate rather than a measured within-study comparison.

Authors’ abstract

Introduction. Clinical and Translational Science Award (CTSA) programs must document their scholars' research impact, but assembling each scholar's record by hand takes staff an estimated 15 hours and does not scale to a full cohort. An artificial intelligence (AI) agent could serve as a tool to gather scholar data across platforms and disciplines. Methods. We built a human-in-the-loop AI agent that assembles a dossier of sourced evidence for each scholar and drafts one-sentence Translational Science Benefits Model (TSBM) impact summaries for staff review. We evaluated it in the impact-reporting workflow of one CTSA hub across 10 career-development (KL2/K12) scholars. Two evaluation staff independently coded all 507 findings as accept, edit, or reject; the primary measure was the unanimous usable rate, defined as the share both accepted or edited. Results. Both reviewers accepted or edited 81.7% of the agent's findings. Reviewers each spent a median of 14 minutes per scholar, replacing an estimated 15 hours of manual assembly. Inter-rater agreement was moderate (Cohen's kappa 0.43 on the usable-versus-reject decision). A profile discovery study found the agent's recall close to human search. The agent's impact evidence spanned all four TSBM domains, and about a third of the reviewed findings fell in non-scholarly categories that routine processes tend to miss. Reviewers rated synthesis accuracy 4.5 and usefulness 4.8 on a 5-point scale. Conclusions. A human-in-the-loop AI agent can serve as the first-pass author of a scholar's impact record, shifting staff from collecting and writing to reviewing, and making cohort-scale impact reporting feasible.