RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents

AI summary

Overview

• Research area: Evaluation of large language model (LLM) agents that operate a command-line interface (CLI) — spanning NLP agent benchmarking, human–computer interaction (HCI) principles for AI systems, and software/ops automation.
• Technical level: Advanced. The paper assumes familiarity with agent scaffolds ("harnesses"), containerized execution environments, environment-state verifiers, and multi-dimensional rubric scoring.
• Scope (one sentence): The paper introduces REALCLI, a paired benchmark of 101 realistic CLI tasks that scores, on identical tasks and environments, both whether an agent finishes the job and how well it collaborates with a scripted human user across five dimensions.

What This Paper Is About

Existing CLI benchmarks hand an agent a complete, fixed specification, let it run alone, and grade only the final code or environment state — so they never measure whether the agent kept the user informed, accepted redirection, admitted uncertainty, or asked before doing something destructive. The authors build a benchmark where the same task can be run either fully autonomously or with a simulated user who releases requirements piece by piece, corrects the agent, revises constraints, and grants or denies authorization. The goal is to test the widely assumed but rarely checked claim that stronger task completion implies better human–agent collaboration.

Key Contributions

1. A five-dimension collaboration framework. Understandability (U), Steerability (S), Observation and Recovery (OR), Trust Calibration (TC), and Safety and Risk Awareness (SR), grounded in HCI theory and derived from anonymized real CLI interaction logs. It is operationalized into 22 subdimensions, a set of interaction scenario templates, and six trajectory-level checks, giving each behavior an explicit expected behavior and required evidence.
2. A paired benchmark, REALCLI. 101 tasks across 10 top-level CLI categories and 52 subdomains, with 1,658 evaluation checklist items, delivered in two modes that share task goal, environment, and verifier: REALCLI-EXEC (full specification upfront, autonomous) and REALCLI-COLLAB (scripted user with progressive information slices and interaction probes).
3. A script-driven user simulator. Rather than a free-form LLM user, each task carries a script specifying information slices, probe trigger conditions, expected behaviors, and evidence requirements, so user turns adapt to the execution trajectory while task semantics and grading criteria stay fixed and reproducible.
4. A joint metric and a 33-configuration study. Task Success, Collaboration Quality, and a multiplicative Joint Collaboration Score (J_collab = R_collab × S_collab), applied to 11 frontier models × 3 agent scaffolds (Claude Code, Terminus-2, Codex).

Main Findings

• Frontier agents score low overall. The highest average score is Claude Opus 5 at 49.2% on Table 3's average column, followed by DeepSeek V4.1 Flash at 45.6%. (The Introduction states the best configuration reaches an average of 52.7%; the abstract and Table 3 report 49.2% — the paper is internally inconsistent here.)
• Joint scores collapse. Peak J_collab is 37.6%, achieved by Claude Opus 5 on Terminus-2, whose R_collab there is 62.4% and S_collab 60.2%. Nearly every configuration shows a large negative gap between autonomous task reward and joint score — e.g., GPT-5.6 Sol on Codex has R_exec 58.5% but J_collab 8.5%, a drop of 50.0.
• User interaction is not a reliable gain. Across 11 Claude Code configurations and 101 matched tasks, interactive vs. autonomous mean pass rates are essentially tied (46.8% vs. 47.4%), but the aggregate hides heavy churn: 17.9% of runs flip failure into success and 18.5% flip success into failure.
• Gains depend on scaffold and task type. Models improve more consistently under Codex (mean gain 3.4 pp). Gains concentrate in tasks with local, verifiable requirement additions (e.g., Data Analysis, Software Engineering); regressions cluster where new instructions must be reconciled with existing constraints, which models often misread as replacing the goal.
• Adaptation strategy, not raw capability, predicts the sign of the change. Qwen3.8 Max gains +15.7% on Codex and Kimi K3 +11.9% on Terminus-2; these models solicit clarification and fold in corrections. Losers act on partial information. On streamlit-headless-app-test, Kimi K3 gains 8.9 pp by treating late requirements as persistent constraints, while Hy3 treats them as isolated API additions.
• Capability is badly imbalanced across the five dimensions. Scores are relatively high on Understandability and Observation & Recovery, and consistently weak on Steerability, Trust Calibration, and Safety. GPT-5.6 Sol on Terminus-2 is the extreme case: R_collab 47.5% but S_collab 19.1%, with Trust Calibration at 7% and Safety at 0%.
• Agents react to blockers better than they track goal expansion. Pass rate for updating a plan after an obstacle is 80.2%, versus 51.7% for recognizing scope expansion. Models pass 98.8% of successive-requirement-change scenarios but only 54.4% for retaining earlier constraints and 29.3% for handling conflicting instructions across turns — the core weakness is cross-turn goal maintenance, not immediate response.
• Outcome-only grading hides collaboration failures. Among successful interactive trajectories, 91.0% still fail at least one applicable collaboration check.
• Dialogue compliance does not reach the artifact. In constraint-retention failures, the final artifact mismatches the stated constraint in 91.3% of cases; among those mismatches with applicable communication checks, 53.3% pass every communication check.
• Verification errors dominate in both modes. Functional or boundary-case errors account for 62.7% of autonomous failures and 53.8% of interactive failures.
• Template-count discrepancy: the abstract and contributions cite 51 interaction scenario templates, while Table 2 and Appendix B list 57 (12 + 11 + 9 + 12 + 13) alongside the 22 subdimensions and six trajectory-level checks.

Methodology in Plain English

The authors first mine real, anonymized CLI session logs, compressing raw traces into task goals, environment states, constraints, and failure signals, then turning that evidence into reusable "skills." Those skills seed synthetic but reproducible tasks, each with its own containerized environment, reference solution, and automatic verifier. A multi-agent rollout stage separates genuine task problems from infrastructure flakiness, routing real problems back for repair, and a difficulty profiler keeps medium and hard instances (easy ones are augmented and recycled).

Each surviving task then gets an interaction layer. Six collaboration scenarios are assigned per task from the template library, and a script is authored: an opening message that deliberately withholds information, information slices released on demand or proactively, and probes carrying a trigger condition, an expected behavior, and checklist evidence. Probes are rendered as ordinary user messages and rewritten if they leak what is being tested. Scripts pass static checks, semantic review, an end-to-end pilot where a real agent runs the task and a judge inspects each scenario, and finally human expert review.

Scoring is deliberately harsh. At the task level, a scenario passes only if all its checklist items are satisfied. At the test-set level, a scenario must be passed in at least 75% of the tasks where it was triggered to count as passed. These binary values are averaged upward through subdimensions, then dimensions, then the five dimensions equally, yielding S_collab. J_collab multiplies S_collab by the collaborative task success rate, so neglecting safety or communication cannot be offset by finishing the job.

All 101 tasks are run in strictly isolated containers, capped at 600 turns, with 6 hours for EXEC and 10 hours for COLLAB, and every reported score uses avg@3 to damp trajectory randomness.

Why This Matters

• Research impact: Shifts CLI agent evaluation from outcome-only grading to a two-axis measurement (does it finish, and does it collaborate), and shows that the two axes diverge sharply — a result that cannot be recovered from pass-rate tables alone. The paired EXEC/COLLAB design also gives a controlled way to isolate the causal effect of user participation, and the strict 75% consistency threshold targets stable behavior rather than lucky single runs.
• Real-world applications:
  • Production incident response and on-call operations, where an agent must stop and ask before running destructive commands or exfiltrating sensitive files.
  • Regulated data processing and analytics, where an agent must confirm scope changes rather than silently re-interpreting the task.
  • Software maintenance and refactoring, where mid-flight requirement changes must persist as constraints across the whole codebase, not just the next reply.
  • Developer tooling/IDE assistants that need to report progress, surface root causes instead of raw logs, and declare capability limits honestly.
• Industry relevance: The finding that identical task success rates can mask radically different collaboration profiles (GPT-5.6 Sol passing roughly half the tasks while scoring 7% on trust calibration and 0% on safety) is a direct procurement and deployment risk for anyone shipping CLI agents. The asymmetry — strong understandability, weak steerability and safety — points to a gap that standard RLHF-style training does not close, and the 91.0% "successful run still fails a collaboration check" figure argues for auditing beyond acceptance tests.

Future Directions

• Close the cross-turn goal-maintenance gap. Agents handle each successive request but lose earlier constraints (54.4%) and conflict resolution (29.3%); training and architectures that treat feedback as persistent constraints on a shared state are the obvious next target.
• Rebuild safety and trust calibration. With Safety scores at 0% for several configurations, an open question is whether better safety can be trained without the steep task-reward penalties currently observed.
• Explain why interaction helps only some models. The paper identifies overconfidence versus proactive clarification as the discriminator, but the mechanism by which user feedback mitigates long-context degradation and escapes local optima is not resolved.
• Extend and verify the framework. Beyond the reported 101 tasks, 10 categories, and 52 subdomains, the framework would benefit from scaling scenario coverage, reconciling the 51-versus-57 template count, expanding to multi-user or real-human settings rather than scripted users, and building training-time methods on top of the benchmark.

Target Audience

Agent researchers and benchmark designers working on tool-use and long-horizon execution; HCI and human–AI collaboration researchers interested in operationalizing concepts like steerability and trust calibration into measurable rubrics; evaluation and safety teams at organizations deploying coding or ops agents; and engineering leaders who need to judge whether a CLI agent will ask before doing damage rather than merely finish the ticket.

Authors’ abstract

Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks. To comprehensively measure collaboration, REALCLI builds a five-dimensional framework, operationalized into 22 sub-dimensions and 51 interaction scenario templates. REALCLI provides two modes for paired comparison: REALCLI-EXEC supplies complete specifications upfront, whereas REALCLI-COLLAB introduces a scripted user that delivers progressive information, corrections, constraint revisions, and authorization decisions. Evaluations reveal profound capability gaps in frontier agents: the best configuration attains an average score of only 49.2%, and when evaluated by the joint collaboration score ($J_{\mathrm{collab}}$), the peak performance reaches merely 37.6%. These results show that task effectiveness diverges sharply from collaboration quality, with models consistently struggling in steerability, trust calibration, and safety boundary management. REALCLI provides a rigorous, reproducible foundation for measuring and advancing collaborative CLI agents.