How can we assess human-agent interactions? Case studies in software agent design

AI summary

Overview

• Research area: Evaluation methodology for LLM-powered agents, human-agent interaction, and human-centered machine learning; instantiated in software engineering agents.
• Technical level: Intermediate. The paper combines a statistical framework (prediction-powered inference) with a large-scale deployed user study; the framework description is accessible, but the estimator derivation assumes familiarity with confidence intervals and A/B testing.
• Scope: The paper proposes an evaluation framework (PULSE) for measuring how agent design choices affect real user satisfaction, and applies it in a deployment with over 15k users of the open-source coding agent OpenHands, then compares those results against 7 code-related benchmarks.

What This Paper Is About

Agent benchmarks almost always assume full automation: an agent completes a well-specified task with no human in the loop. Real use, by contrast, is collaborative, with users supervising, correcting, and rating the agent's work as it unfolds. The paper asks how to rigorously measure the effect of agent design changes on human satisfaction, and introduces PULSE (Prediction-powered User Label Synthesis and Evaluation), a three-step framework that combines sparse human ratings with model-generated predictions to produce valid, tighter effect-size estimates. The authors then run three case studies on a deployed software engineering agent to see which design choices actually matter to users, and whether benchmark rankings agree with user preferences.

Key Contributions

1. The PULSE framework. A three-step protocol for human-centric agent evaluation: (1) collect user feedback through an interface designed around "work segments," (2) train an ML model to predict user satisfaction from interpretable interaction features so that unrated sessions can be imputed, and (3) extend prediction-powered inference (PPI) to produce valid confidence intervals on the effect size of an agent design change.
2. A large-scale in-the-wild deployment. The framework is instantiated in a web platform built around the open-source agent OpenHands, spanning over 36k sessions from 15k different users, with N = 1747 labeled trajectories (mean rating 4.07) and roughly 20 times more unlabeled sessions, since only about 5% of interactions receive ratings.
3. Three case studies varying agent design. Separate studies on the LLM backbone (claude-3.7-sonnet, claude-4-sonnet, gpt-5), a planning mechanism (a task_tracker tool surfacing TASKS.md to the user), and memory management (lowering max_step from 120 to 80).
4. Benchmark comparison. An exploratory analysis relating changes in human ratings to changes on 7 code-related benchmarks, finding that benchmark improvements do not reliably translate into user preference.

Main Findings

• PULSE tightens confidence intervals. Confidence interval bands decreased by an average of 39.5% across the 4 experiments compared to a standard A/B test, described in the abstract as a reduction of 40%. In the claude-3.7-sonnet versus claude-4-sonnet comparison, the 95% CI for the naive estimator was [-3.95%, 11.78%], while the augmented-label CI was [1.99%, 8.06%].
• Model backbone matters more than scaffolding. Users significantly preferred agents powered by claude-4-sonnet over both other LLMs: a 5.86% difference in user satisfaction between claude-3.7-sonnet and claude-4-sonnet, and a -7.83% difference between claude-4-sonnet and gpt-5. Scaffolding changes produced a smaller but significant 3.1% difference for planning versus no plan. Overall, the paper reports model changes at Δ = 6-8% and scaffold changes at Δ < 3%.
• Showing the agent's plan helps. Despite only a small rating improvement, the no-plan variant showed the agent was 12.8% more likely to misunderstand the user, with corresponding increases in insufficient analysis (13.0%) and insufficient debugging (14.4%).
• Behavioral features explain the gpt-5 gap. Trajectories with gpt-5 contained 32% fewer user messages on average, and there were 16% fewer code pushes, suggesting users disengaged and abandoned prompting earlier.
• Memory reduction can save cost without hurting experience. Lowering max_step from 120 to 80 was estimated to yield an amortized 0.5 cent saving per step, based on simulations on SWE-Bench, without degradation to user experience.
• Learned feature-based predictors beat LLM-as-a-judge. On MSE, MAE, and correlation, models trained on 15 extracted features outperformed long-context LLM judges (o3, gemini-2.5-pro, claude-4). Random forests achieved correlation 0.29 ±0.01 versus 0.22 ±0.03 for o3, 0.14 ±0.07 for gemini-2.5-pro, and 0.23 ±0.01 for claude-4; the paper states predictive methods improve correlation with outcomes by at least 26% over the baseline. LLM judges also tended to be more pessimistic than users (for example, o3 rarely gave a score of 5).
• The most important features are user sentiment and git push. These combine an interaction signal with a task-completion signal, though no single feature is fully predictive of user rating.
• Human preferences and benchmarks diverge. On the claude-4-sonnet versus gpt-5 comparison, the Pearson correlation between the human-rating difference and the benchmark difference was ρ = -0.18 (the body text states -0.11), versus ρ = 0.66 for the claude-3.7-sonnet versus claude-4-sonnet comparison. The abstract notes that gpt-5 outperforms claude-sonnet-4 on 6 out of 7 benchmarks, while humans preferred claude-sonnet-4 on 4 out of the 7 task subsets. Alignment was strongest for testing code benchmarks and administrative tasks, and the largest human-rating difference came from fixing continuous integration issues rather than standard code-base work.
• Ratings weakly but positively align with objective actions. User satisfaction correlated positively with git push (r = 0.117, p < 0.001) and git commit (r = 0.101, p < 0.001), and near zero for other git actions.
• Labeled and unlabeled trajectories are broadly similar. Comparing all 15 features with rank-biserial correlation, only user message count showed a moderate difference (RBC = 0.32); all other differences were small or negligible (RBC around 0.1 or less).
• Feedback is sparse but diverse. Only 12.75% of users contributed ratings in multiple sessions, and most of those contributed only 2 sessions. The user base was diverse: Python was the most popular language (29.52%), users were predominantly English-speaking (82.61%), and users sent a median of 10 messages per session. A manual inspection of 20 low-rated sessions found failures such as failed tests/CI, missing dependencies, and port/health-check failures driving user frustration.

Methodology in Plain English

The authors do not ask users to rate every single interaction; that would be invasive and would still yield few labels. Instead, they prompt for a 5-star rating at the end of each "work segment"—the span between a user sending a command, the agent running, and the agent stopping. A session can contain several segments, and multiple ratings within a session are averaged.

Because only about 5% of interactions get rated, the team extracts 15 interpretable features from each session in three categories: features about the user (message content, sentiment, number of messages), features about the agent (task category such as implementing features or fixing bugs, and failure modes such as insufficient testing or did not follow instruction), and features showing task progression (git actions like commit, push, pull, reset, rebase). Features are detected either by an LLM judge (using a prompt with gpt-5-mini) or by analyzing the event stream. These features are then used to train models ranging from logistic regression to random forests that predict the missing satisfaction ratings.

To compare two agent variants, the authors use A/B testing with randomization at the conversation level, collecting at least 150 labels per condition and running each comparison for 2-3 weeks, with a significance cutoff of α = 0.05. The naive estimate is just the difference in mean ratings. The PULSE estimate extends prediction-powered inference: it starts from the labeled sample mean and adds a correction term that compares the model's predictions on the large unlabeled pool against its predictions on the labeled pool, with a tuning parameter chosen to minimize variance. Because this correction absorbs systematic model error, the prediction model need not be accurate or unbiased—only the labeled and unlabeled data need to come from the same distribution. The final step forms a Wald confidence interval on the difference between the two conditions.

Why This Matters

Impact on research. The paper argues that benchmark-driven evaluation, which assumes full automation, is insufficient for agents that are actually used collaboratively. It provides a reusable statistical recipe (PPI-based effect estimation) for running human-in-the-loop experiments at a fraction of the labeling cost, and it empirically demonstrates a case where benchmark rankings and human preferences point in opposite directions. It also contributes a comparison table (Table 1) positioning prior work: static benchmarks and interactive benchmarks vary agent design but use no real users and no in-the-wild tasks; prior human studies use real users but do not vary agent design; this work claims all four properties.

Real-world applications.

• Developer tooling teams choosing between LLM backbones for a coding assistant, where the paper's results suggest backbone choice dominates scaffolding tweaks in user-perceived quality.
• Interface design for agentic coding products, where displaying the agent's plan (via TASKS.md) reduced misunderstanding of user intent.
• Cost management for deployed agents, where adjusting context/memory thresholds (lowering max_step from 120 to 80) was estimated to save an amortized 0.5 cent per step without harming experience.
• Evaluation platforms in other human-agent domains, since the authors position PULSE as a general framework for evaluating human-agent collaboration, not just software engineering.

Industry relevance. The study is conducted on a deployed open-source product (OpenHands) with over 15k users, so the signal is about real usage rather than curated tasks. The finding that a model can lead on 6 of 7 benchmarks yet be less preferred by users on 4 of 7 task subsets is a direct caution for teams that select models or ship changes based on benchmark leaderboards alone. The authors also note that methods enabling more effective agents may increase automation of knowledge work, with implications for labor, job roles, and skill requirements.

Future Directions

• Apply PULSE beyond software engineering. The authors explicitly call for applying the framework to additional agents and domains to understand how design insights transfer, noting the current case studies are limited to a single platform (OpenHands).
• Model label noise. The paper uses user ratings as the primary human-centered metric but does not explicitly model label noise; future work should explore richer measures of agent performance and noise-aware statistical inference.
• Automate and broaden feature discovery. The appendices explore using LLMs without a human in the loop to automatically discover most of the 15 features, suggesting a path toward scaling the feature engineering process.
• Move beyond SWE-Bench-style evaluation. Because the largest human-rating differences came from continuous integration fixing and testing rather than standard bug fixing, the authors argue for expanding the benchmark landscape even within software engineering.
• Reproducibility and data release. Raw code contexts cannot be shared for privacy reasons, so the authors plan to release the PULSE framework code, extensions to the OpenHands platform, and anonymized feature-level datasets.

Target Audience

Researchers and practitioners in LLM agent evaluation, human-computer interaction, and applied machine learning who need to measure the effect of system changes on real users; product and platform engineers building coding agents or similar human-in-the-loop AI tools; and statisticians or methodologists interested in applying prediction-powered inference to noisy, sparsely labeled human feedback. Readers seeking a purely benchmark-oriented or fully automated evaluation recipe will find the paper's central argument is that such recipes are insufficient on their own.

Authors’ abstract

While benchmarks measure the accuracy of LLM-powered agents, they mostly assume full automation, failing to represent the collaborative nature of real-world use cases. In this paper, we make two major steps towards the rigorous assessment of human-agent interactions. First, we propose PULSE, a framework for more efficient human-centric evaluation of agent designs, which comprises collecting user feedback, training an ML model to predict user satisfaction, and computing results by combining human satisfaction ratings with model-generated pseudo-labels. Second, we deploy PULSE in software engineering -- one of the highest-impact, real-world domains for LLM agents -- via a large-scale web platform built around the open-source agent OpenHands. Across 15k users, we evaluate how three agent design decisions impact developer satisfaction rates. We also show how PULSE can lead to more robust conclusions about agent design, reducing confidence intervals by 40\% compared to a standard A/B test. Finally, we find substantial discrepancies between in-the-wild results with benchmark performance (e.g., the anti-correlation between claude-sonnet-4 and gpt-5), underscoring the limitations of benchmark-driven evaluation. Our framework PULSE provides guidance for future evaluations, and our findings identify opportunities for better software agent designs.