Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows

AI summary

Overview

Research area: Human–AI collaboration, agentic benchmarking, and multimodal LLM evaluation.

Technical level: Intermediate. The paper is readable without deep technical background, but familiarity with benchmark design (F1, MAE, Spearman correlation) helps.

Scope: Kolab is a dataset of 115+ hours of screen recordings from 50 professionals in 40 occupations working with AI agents on their own long-horizon, real-world problems, paired with step-level human judgments used to benchmark whether frontier multimodal models can anticipate and evaluate collaboration.

What This Paper Is About

Most agent benchmarks ask whether an AI can finish a task autonomously in one attempt. Real professional work does not look like that: goals shift, early decisions constrain later ones, and people adopt, edit, or discard what an agent produces, often outside the chat window entirely. The authors build Kolab to test a different capability — whether a model can tell how a professional actually used its contribution, and how satisfied that professional was with the collaboration, purely from the unfolding workflow.

Key Contributions

1. A new dataset of real collaborative workflows. Kolab contains 115 hours of screen recordings from 50 professionals across 40 occupations (10 O*NET sectors), solving their own problems in at least five interdependent steps. It comprises 100 workflow videos and 508 steps spanning 90+ softwares, with each workflow ending in a concrete deliverable and step-level human ratings of collaboration experience and output quality.

2. A graph representation of collaboration with adoption labels. Each workflow is modeled as a temporal graph of human and agent action nodes and their dependencies, annotated with whether the human Adopt (65.9%, 456 instances), Modify (19.1%, 132), or Reject (15.0%, 104) each agent output — 692 labeled instances in total.

3. Three metrics for describing collaboration dynamics. Collaboration burstiness (from inter-event intervals), workflow topology (branch nodes with out-degree > 1 relative to the number of edges), and AI retention rate (the share of agent outputs adopted or modified).

4. A benchmark of two untested capabilities. Task RQ1 asks models to recognize (1a) and forecast (1b–1d) how professionals use agent contributions; RQ2 asks models to predict 1–5 human ratings of collaboration satisfaction (2a) and output quality (2b). Both are evaluated across seven combinations of video, graph, and chat inputs.

Main Findings

• Forecasting is hard even for the strongest models. With video only, Gemini 3.1 Pro reaches 0.391 F1 and GPT-6 reaches 0.513 F1 at forecasting adoption before the human responds. Recognition (with the human response visible) is easier: Claude Opus 5.5 reaches 0.676 F1 and GPT-6 reaches 0.713, though GPT-6 was evaluated on fewer instances (279 for 1a, 349 for 1b, versus 692) due to cost limits.

• Claude's advantage is largely one label. Because the study's default agent was Claude, the authors hypothesize Claude Opus 5.5 benefits from understanding its own interface, but its forecasting F1 of 0.492 is dominated by Adopt (0.790), with much weaker Modify (0.434) and Reject (0.250).

• Recovering human ratings fails against a trivial baseline. The mode baseline beats every model on MAE: 0.682 for collaboration satisfaction and 0.735 for output quality, versus the best model (Gemini 3.1 Pro) at 0.685 and 0.764. Rank agreement is weak: GPT-6 has the highest Spearman correlations at ρ = 0.379 for collaboration satisfaction and ρ = 0.352 for output quality, while several models sit near zero.

• Longer context does not reliably help. Extending context to 1–3 preceding steps (Experiment 1c) or the full workflow (Experiment 1d) on 114 instances changes macro-F1 by less than 0.07 for most video and graph settings, and no video-based improvement was statistically significant. Chat logs benefit most (Gemini 3.1 Pro gains 0.12 F1 with three preceding steps, p < 0.01), but this does not persist with the full workflow; Qwen-32B drops from 0.43 to 0.32 F1 with the full graph (p < 0.01).

• No single modality wins. Video alone gives the best recognition for GPT-6 (F1 = 0.71); graphs, alone or combined, tend to give the best forecasting (GPT-6 at 0.52). Chat alone hurts Reject recognition (change of −0.07 F1 relative to video, versus +0.07 for Adopt and +0.01 for Modify), likely because chat does not show what happens outside the AI interface. Graphs help the non-agentic Qwen models most: Qwen-32B's forecasting F1 rises from 0.23 to 0.37 when graphs are added to video.

• Models carry strong label priors. Gemini 3.8 Flash predicts Adopt for 83.8% of examples against a 65.9% ground-truth rate; Qwen3-32B predicts Modify in 87.3% of cases against 19.1% in ground truth. Gemini 3.1 Pro labels 52% of true Modify and 71% of true Reject cases as Adopt.

• Rating predictions collapse toward the positive end. With video, Gemini 3.1 Pro predicts 4.83 for a true rating of 1 and 4.95 for a 5 in Experiment 2a (5.00 and 4.94 in 2b). Claude's predictions range only from 3.3 to 3.9 across the whole 1–5 scale, and Qwen variants from 3.04 to 4.15. Graph input improves separation, with gaps between true ratings of 1 and 5 reaching 1.95 and 2.45.

• Collaboration rhythm tracks independent planning. Burstiness was consistent within individuals across two sessions (Spearman ρ = 0.59). Burstier workflows had longer solo lead-ins (ρ = 0.35, p < 0.001), and participants who reported having a clearer plan before using AI also had longer solo lead-ins (ρ = 0.53, p < 0.001).

• Concurrent work comes with a cost. Workflow branching is negatively associated with collaboration ratings (ρ = −0.24, p < 0.05).

• Retention varies by profession. Adoption ranges from 57% in Legal to 79% in Sciences and Computer & Mathematical work. Legal stands out on modification at 41%, roughly twice the next-highest domain (Arts & Sports at 22%).

Methodology in Plain English

The researchers recruited 50 professionals through word of mouth, Prolific, and Upwork, paying $100 each with a bonus for extra time. To be included, a problem had to come from the participant's actual work, require genuine domain expertise, need at least an hour and five interdependent steps, and produce a concrete deliverable.

Each participant planned their workflow in advance, then worked with an agentic AI (Claude, by default) across two roughly one-hour screen-recorded sessions, free to use other tools such as ChatGPT, Gemini, or Grok where appropriate. Afterwards they rated their collaboration experience and output quality on a 1–5 scale and completed a survey.

The team then converted each step into a temporal graph of human and agent actions with timestamps, refined jointly with the professionals. Two annotators identified source nodes (the human request that triggered the agent), target AI nodes (the agent's output), and the human node that acted on that output, labeling it Adopt, Modify, or Reject.

For the benchmark, models were shown screen video spanning from the delegation request up to different endpoints — including or excluding the human's response — and asked to predict the adoption label; for the rating tasks they watched the full step and predicted the professional's 1–5 scores. Six models were tested: Gemini 3.8 Flash, Gemini 3.1 Pro, GPT-6 Astra, Claude Code Opus 5.5, Qwen3-VL-30B-A3B-Instruct, and Qwen3-VL-32B-Instruct, against majority-class and mode-rating baselines, over three runs each.

Why This Matters

Research impact. The paper argues that strong performance on autonomous agent benchmarks does not imply effective real-world collaboration, and it supplies a public benchmark for the missing capability: modeling a human collaborator rather than completing a task. By grounding evaluation in naturalistic screen recordings with human-curated graphs and step-level ratings, it offers a process-level signal that reference-answer benchmarks cannot provide.

Real-world applications:

• Process reward models. If a model could recover expert ratings from a workflow alone, those judgments could train collaborative agents without asking experts to label anything.
• Deployment monitoring. An automatic monitor could flag friction or low-quality outcomes in deployed systems as they occur.
• Scalable evaluation of professional work. Collaboration quality could be assessed on tasks where no reference answer exists, such as legal drafting or architectural layout.
• Proactive agent behavior. A model that forecasts likely rejection could ask a clarifying question at delivery time, flag parts it expects to be edited, or escalate when errors would propagate to later steps.

Industry relevance. The dataset deliberately emphasizes non-software-engineering professions — film editors, graphic designers, audio engineers, aerospace engineers, GIS specialists, attorneys, bioinformaticians — and covers 90 distinct profession-specific non-AI softwares plus 9 AI tools. That breadth matters for vendors building assistants that must interoperate across many applications and even across competing AI models, since professionals in the study mixed Claude with Gemini, ChatGPT, and Grok.

Future Directions

• Training data and sandboxes. The authors propose constructing training datasets or sandbox environments where models practice collaborating with simulated users before real-world deployment.
• Better context use. Since longer workflow context and additional modalities do not consistently improve predictions, it remains open what form of memory or representation a long-running agent actually needs to model its collaborator.
• Escaping label priors. Models systematically overpredict Adopt and overpredict high ratings; how to make predictions sensitive to observed human behavior rather than model-specific tendencies is unresolved.
• Cross-model understanding. Whether an agent can learn to read human collaboration with other AI systems — not just its own interface, where Claude showed an advantage — is an open question the authors raise directly.

Target Audience

Researchers and practitioners working on agentic AI, human–AI interaction, and benchmark design; teams building collaborative assistants or computer-use agents that must operate over long sessions; and evaluators who need process-level signals for professional tasks without reference answers. It is also useful for readers in the specific professions represented, since the reported workflows and adoption patterns come directly from practitioners in law, architecture, music production, and video production, among others.

Authors’ abstract

When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place greater importance on ongoing interaction with AI agents (e.g., revising plans) than on an agent's ability to complete the task in a single attempt. However, most agentic benchmarks have focused more on automation. Strong performance on such benchmarks does not imply effective real-world collaboration. Professional workflows offer a useful setting, as they often involve interdependent steps. To address this gap, we introduce Kolab, a new dataset of 115+ hours of screen recordings of 50 professionals from 40 occupations collaborating with agents on realistic, long-horizon problems. Kolab also contains participants' collaboration and output quality judgments. Our analysis reveals diverse collaboration patterns, highlighting the importance of when and how professionals coordinate with agents. A collaborative agent must do two things that automation benchmarks never test: anticipate whether experts will adopt, modify, or reject its contribution once it leaves the chat window to guide its next steps, and assess collaboration quality from the unfolding workflow without constant expert feedback. Our benchmark operationalizes both: (1) forecasting how professionals use agent contributions at diverse workflows, and (2) verifying their judgments of collaboration satisfaction and output quality. Findings show that frontier models, both agentic and non-agentic, overestimate collaboration with experts and struggle to distinguish between high- and low-quality collaboration. Longer workflow context does not consistently improve these predictions. As frontier agents cannot yet understand long-horizon collaboration with experts, we position Kolab as a valuable dataset for studying human-agent collaboration grounded in real professional workflows.