RunningTab: Direct Workspace Interaction with Environment-Side Tabs

AI summary

Overview

• Research area: LLM agents that operate a computer, specifically AI agents performing knowledge work over file workspaces (agent memory, agentic search, and "direct corpus interaction").
• Technical level: Intermediate to Advanced. It assumes familiarity with LLM agents, terminal tool use, context-window limits, and retrieval concepts such as BM25, but the central idea (an environment-maintained task record) is intuitive.
• Scope: The paper introduces an "environment-side tab" — a per-task record kept by the execution environment rather than the model — and evaluates it on three benchmarks with three LLMs against plain direct workspace interaction and model-kept tracking baselines.

What This Paper Is About

Many workplace tasks produce a new deliverable (a report, a spreadsheet, a review) from files an organization already holds, and LLM agents are starting to take this work over by searching and reading those files directly from a terminal — a setting the authors call direct workspace interaction (DWI). The problem is that reaching the files is only half the task: what the task asks for, what has been read, and which files were listed but never opened all pass through the context window and leave no trace, so an agent can open the right annual report, extract a figure, and still deliver a report without that figure. The goal of RunningTab is to supply that missing record by keeping a running, per-task tab of what the task still owes beside the agent, in the environment.

Key Contributions

1. A diagnosis of a specific, previously unaddressed failure mode. The authors analyze DCI (plain direct corpus interaction) with GPT-5.4 nano on Workspace-Bench and show that extraction failures — where the agent opened the right files but the deliverable lacks or distorts their content — account for 51.1% of lost rubric points, split into "Shown but Missing" (12.3%) and "Dropped or Distorted" (38.9%), with discovery failures adding 17.6%. This is a loss not covered by prior diagnoses that focus on document understanding or lineage tracing.
2. The environment-side tab. A per-task object defined as T_t = (R, L, C): requirements R stated by the agent, a read log L captured by the environment (each entry storing an excerpt exactly as shown to the model plus an anchor of file path and command), and candidates C of files listed but not opened. It is kept by the environment and stored outside the context window and outside the workspace.
3. Four tab operations plus three settling mechanisms. Agent-side operations are add and resolve; read operations are list and show. Settling is supported by a BM25-ranked review that shows each open requirement beside best-matching read-log entries and top unopened candidates, an evidence check that refuses a "done" citation whose path and excerpt share no term with the requirement, and a finish check issued at the agent's stop.
4. Empirical validation across three benchmarks and three LLMs. RunningTab achieves the best result on every benchmark, metric, and LLM tested, plus ablations attributing most of the gain to what the environment captures on its own.

Main Findings

• Consistent wins on all benchmarks and models. RunningTab beats DCI and the model-kept baselines (TODO, Self-Refine, Self-Tracking) on every metric for GPT-5.4 nano, DeepSeek V4 Flash, and Gemini 3.8 Flash. Examples: with GPT-5.4 nano on Workspace-Bench, Pass Rate rises from 41.63 for DCI to 44.73 and TCR@70 from 20.67 to 25.67; with DeepSeek V4 Flash, Pass Rate goes from 53.28 to 59.99; with Gemini 3.8 Flash, Pass Rate goes from 31.42 to 46.91 and OfficeQA Pro Acc@1% from 45.64 to 62.82.
• Model-kept tracking is not the source of the gain. The baselines in which the model itself maintains the record remain far less effective, suggesting the benefit comes from a record the environment keeps on the agent's behalf, outside the context window.
• The tab holds what was shown but missing. Among numeric values a failed rubric expected, that appeared in an observation, and that are absent from the deliverable, the tab holds 94.8% with GPT-5.4 nano. With the task and requirement given but values masked, the same LLM recovers 8.4% of held values without the tab, 45.8% from the review for that requirement and the entries it opens, and 78.1% from the whole read log. RunningTab also succeeds on more "Shown but Missing" rubrics than DCI (27.0% against 20.7%).
• Ablations attribute the largest drop to environmental capture. On Workspace-Bench with GPT-5.4 nano, removing environmental capture drops Pass Rate to 41.36 (a fall of 3.37), essentially the DCI level of 41.63; removing the review costs 2.86; removing the evidence check costs 1.84; removing the finish check costs 1.84. All four components lower both Pass Rate and TCR@70 when removed.
• Gains grow with reading volume. Splitting Workspace-Bench tasks into thirds by how much the agent reads, the gain over DCI generally grows from lightest to heaviest third across all three LLMs, averaging twice as large on the heaviest third as on the lightest. Reading-heavy tasks are also where DCI performs worst, with its Pass Rate dropping 16 to 27 points from the lightest to the heaviest third.
• Different LLMs operate the tab differently. Gemini 3.8 Flash works mainly through the review (stating requirements and pulling the review before the finish check in every analyzed attempt; citing a reviewed entry for done in 76.9% of attempts). GPT-5.4 nano leans on the finish check (operating the tab after it in 72.6% of attempts and editing the deliverable in 17.1%; using show in 74.6%). DeepSeek V4 Flash works through candidates (reading a median of 11 distinct files and opening a listed candidate in 54.7% of attempts).
• The tab captures a substantial record. On Workspace-Bench, a median of 4 to 5 requirements, 6 to 18 read-log entries spanning 5 to 11 distinct files and 12K to 40K characters of excerpted text, and 147 to 294 listed-but-unopened candidates. When the review is pulled, 69.9% to 88.0% of open requirements appear beside a read-log entry containing words of the requirement.
• The problem is real in plain DWI. About one in five (19.8%) of the 891 distinctive numeric values the rubrics expect and that appear in an agent observation are absent from the deliverable.

Methodology in Plain English

The authors first define direct workspace interaction: a task asks for a deliverable assembled from many files scattered across a workspace, and the agent reaches those files with ordinary terminal tools (read, bash) rather than through an index. They then quantify what goes wrong by running DCI with GPT-5.4 nano on Workspace-Bench over three runs of its 100 tasks, categorizing each failed rubric with LLM annotators (which agreed with human labels at a Cohen's kappa of 0.83) and checking whether each expected numeric value appeared in the agent's observations and in its deliverable.

Their fix changes nothing about how files are reached. Instead, the environment that runs the model and its tool calls maintains a tab next to the task. The agent states its requirements up front through add. The environment, which sees every observation, then decides on its own which files were read (their content appears) versus merely listed (only their path appears), storing reads verbatim as excerpts with provenance and ranking candidates by BM25 lexical relevance of their paths to the task. A status line appended to each observation tells the agent what is still open and how much the tab holds. Before resolving anything, the agent calls list to see each open requirement beside its best-matching excerpts and top unopened candidates. A requirement can be resolved as done only by citing a supporting read-log entry that survives a term-overlap check, or as unavailable with a reason. If the agent stops with items open, it gets one finish check listing what remains.

Evaluation uses DCI-Agent-Lite, a terminal agent built on the Pi harness, with a budget of 300 turns, runtime context management at level 3, and a one-hour limit per task. The three LLMs are GPT-5.4 nano, DeepSeek V4 Flash, and Gemini 3.8 Flash; where an LLM judge is required, the paper follows each benchmark's protocol and uses GPT-5.4 mini. Tab-operation calls are removed from the trajectory that evaluators read, and results are averaged over three runs. The three benchmarks are Workspace-Bench (five professional roles with thousands of heterogeneous files; 100 tasks, 1,850 evaluation instances; Rubric Pass Rate and TCR@70), TheAgentCompany (a simulated software company checked by programmatic evaluators; a subset needing no service other than the company file server, with 88 evaluation instances; Success and Score), and OfficeQA Pro (697 files, U.S. Treasury Bulletins spanning nearly a century, about 89,000 pages; a subset needing no external data source, 104 evaluation instances; Acc@0% and Acc@1%).

Why This Matters

• Research impact. The paper reframes the bottleneck in workspace agents. Prior work points to document understanding and lineage tracing; this work isolates a distinct loss — content the agent already saw, which the context window displaces before delivery — and shows that an environment-kept record addresses it. It also draws a sharp line between environment-kept and model-kept task state, a distinction the authors argue prior agent-memory and multi-agent orchestration systems do not make.
• Real-world applications:
  • HR and recruiting: compiling a hiring summary from resumes and interview notes, the paper's own motivating example.
  • Finance and budgeting: building a budget review from a year of monthly spreadsheets.
  • Proposal and contract work: drafting a project proposal from meeting notes and a previous contract.
  • Enterprise document QA: answering questions from long archival corpora such as the U.S. Treasury Bulletins in OfficeQA Pro, where answers must often be computed from the files.
• Industry relevance. The results suggest a cheap, model-agnostic improvement: three different LLM families all improve when the environment keeps the record, and RunningTab's settings stay the same across benchmarks and models. The ethical statement notes that the tab stores excerpts of files the agent reads, so it may hold sensitive content in real workspaces, and recommends safeguards such as keeping the tab under the same access controls as the workspace and reviewing deliverables before deployment.

Future Directions

1. Reducing dependence on the model to state requirements. The requirements are the one entry the environment cannot capture on its own, since decomposing a task into items takes an understanding the model supplies; the paper notes requirement-stating rates vary widely across LLMs (GPT-5.4 nano states requirements before the check in 56.9% of attempts, versus 100.0% for DeepSeek V4 Flash and Gemini 3.8 Flash).
2. Closing the remaining tab gaps. The read log keeps only file content, not values the agent computes (such as a sum) or sees only in a listing — the authors' own footnote to the 94.8% figure. How to capture computed and listed-only values is left open.
3. Understanding per-model operating styles. Each LLM draws on the tab differently (review-driven, finish-check-driven, candidate-driven), which raises the question of whether tab design should be adapted per model or whether a single design can be made to work uniformly.
4. Testing in larger, more adversarial, and real workspaces. The evaluation covers three benchmarks and three LLMs with a one-hour limit and 300-turn budget; behavior under heavier privacy and access-control constraints in genuine organizational workspaces is not yet studied.

Target Audience

Researchers and engineers building LLM agents for knowledge work, agent-memory and tool-use infrastructure, and enterprise automation. It is also relevant to practitioners designing evaluation harnesses for long-horizon workspace tasks, and to readers interested in the division of labor between a model and its environment — where the environment, not the model, is the party best positioned to keep an accurate record. Readers without background in LLM agents or retrieval will find the core idea accessible, but the benchmark metrics (TCR@70, Acc@0%/Acc@1%, checkpoint-based Success and Score) and the BM25-based ranking details assume some familiarity with agentic evaluation.

Authors’ abstract

Much knowledge work produces new deliverables from files a workspace already holds, and LLM agents are beginning to take such work over. Through direct corpus interaction, an agent can search and read any of those files from a terminal with no indexing, and producing a deliverable from many of them in this way is what we call direct workspace interaction (DWI). Reaching the files, however, is only half the task: nothing keeps track of what the task asks for, what has been read, and what was listed but never opened, all of which slip through the context window without leaving a trace, so an agent may extract a figure and still deliver a report without it. To address this, we present RunningTab, a framework that equips direct workspace interaction with an environment-side tab: a per-task record of what the task still owes, kept by the environment alongside the agent. Specifically, the agent adds its requirements, while the environment records every file read as an excerpt with its provenance and every listed but unopened file as a candidate; the agent can then see each requirement beside its best-matching excerpts and top unopened candidates, resolve it against matching content or set it aside with a reason, and, should it try to finish with requirements still open, receive them in a finish check. We validate RunningTab on three benchmarks with three LLMs, where it consistently outperforms plain DWI and baselines that keep the record in the model, while its tab usually holds the values a deliverable needs once seen.