Dr. Claw: An AI Scientist Workspace for Vibe Research

AI summary

Overview

Research area: AI research automation / human–AI collaboration systems (agent orchestration, AI scientist tooling, HCI for AI-assisted research).

Technical level: Intermediate. The paper assumes familiarity with command-line coding agents (Claude Code, Gemini CLI), agent orchestration concepts (task graphs, checkpoints, human-in-the-loop interrupts), and basic statistical reporting (Friedman tests, bootstrap confidence intervals), but the system design is explained conceptually rather than through low-level implementation detail.

Scope: The paper introduces Dr. Claw, an open-source workspace that wraps existing command-line coding agents in a controllable, auditable human-in-the-loop research workflow, and reports a small exploratory pilot evaluation plus a scenario walkthrough.

What This Paper Is About

Command-line coding agents can already read and write files and sustain long sessions, but end-to-end research still fragments across chat tools, IDEs, terminals, and writing environments, and the decisions that would make a process auditable are rarely preserved. The authors argue the bottleneck is now full-process orchestration rather than isolated capability, so they build Dr. Claw: a workspace that links human decisions to AI execution across planning, execution, and writing in one traceable, recoverable loop. Dr. Claw deliberately does not introduce a new executor — it wraps an existing one and adds the state, control, and audit layer that such agents lack.

Key Contributions

1. Formalizes "Vibe Research." A controllable, human-in-the-loop research-orchestration paradigm that clarifies the boundary between AI execution (high-throughput, parallelizable, templatable work such as retrieval, coding, running, summarizing, drafting) and human decision responsibility (research direction, evaluation criteria, key trade-offs, final acceptance). It is explicitly distinguished from end-to-end autonomous research agents.

2. Implements the paradigm in Dr. Claw. The system combines task-graph-centric orchestration, a chat-driven planner, a modular skill library (58 stage-mapped skills across five research stages, 171 in the deployed catalogue), and a multi-agent execution layer compatible with mainstream coding agents.

3. Provides a controlled pilot evaluation plus a scenario-based demonstration. Holding the backend executor fixed, Dr. Claw scores higher than the bare agent on research completeness while uniquely persisting an auditable, recoverable process trail (consistent across tasks, though not statistically powered at one run per task).

4. Adds a retrospective three-condition human study associating the integrated workflow with gains in efficiency, quality, and usability over non-integrated workflows.

Main Findings

• Completeness advantage is driven entirely by research hygiene, not modeling. Dr. Claw pooled 0.952 of the 21 best-practice elements versus the bare agent's 0.873 (Δ = +0.079), winning two of three tasks and tying the third. Both conditions passed the modeling elements (three-plus models, cross-validation, calibration, ablation, statistical rigor) at 1.00, so the gap lives in hygiene: limitations section (0.33 → 1.00 pooled pass rate), subgroup analysis (0.33 → 1.00), and real literature citations (0.00 → 0.67; the bare agent produced zero citations across the three tasks).

• The result is directional, not significant. With one replicate per task, the pooled completion Δ = +0.079 has a 95% bootstrap CI of [-0.00, +0.14] that includes zero. Dr. Claw remains slower — a bounded overhead the orchestration layer incurs by recording state.

• Skill suggestion is guaranteed; skill invocation is best-effort. Skill selection is deterministic given a task's stage and type, but invocation is not enforced — the resolver can only place a suggestion in the task prompt. The executor read roughly 12 skill files per run, yet on the one tie (the clinical-note task) the reference audit did not fire, so citations were missed. Closing that gap — verifying skill execution rather than recommending it — is described as the clearest reliability improvement the current design admits.

• Auditability is an architectural affordance, not a score. Every Dr. Claw run persisted a queryable task graph (mean 14 nodes), a timestamped execution trace (mean 14 transitions), a decision-log brief (mean 10 entries), and named research stages with explicit claim-to-evidence maps; the bare agent persisted none. The authors state "bare = absent" holds by construction, since these are Dr. Claw's own file format, and claim no superiority on format-neutral traceability (write-up file references resolved at 100% versus 62%, but volume-confounded at 48 versus 8 references and non-decisive at this n).

• Non-destructive failure recovery. On a Derm7pt mini-project with the same codex/gpt-5.4 backend, Dr. Claw halted on an induced wrong-path error rather than silently self-correcting, then recovered to accuracy 0.892 with all 5 pre-existing files retained across 23 tool events. The workspace adds corrected artifacts rather than wiping the failed attempt. This is reported as a single-condition design demonstration, without a matched bare-agent recovery run.

• Human study favors Dr. Claw on all four metrics. Across three stages (Ideation, Experiment, Publication), Dr. Claw was associated with shorter completion-time bands (mainly under 1h and 1h–1d), the highest blind-rated stage output scores (Dr. Claw > Web/Desktop-AI > No-AI on a 1–5 scale), and lower switching bands with higher experience scores. The omnibus signal is strongest and fully pairwise-significant for Experience (χ²(2) = 12.29, p = 0.0021 / 13.56, p = 0.0011 / 12.07, p = 0.0024, with pairwise Holm-corrected p = 0.0469 for both controls at all three stages); Time Band was also significant in Stages 2–3 (13.56, p = 0.0011 and 14.00, p < 0.001).

• The evaluation is exploratory. The authors describe it as a small-scale demonstration rather than a powered comparative study, and note that holding the executor fixed is not an ablation: task graph, state objects, skill library, and workflow instructions arrive as one bundle, so the gap cannot be attributed to any single component.

Methodology in Plain English

The authors first build a design-space comparison table across ten systems (AI Scientist v1/v2, Agent Laboratory, ResearchAgent, AutoGen Studio, Flowise, LangGraph, TinyScientist, ResearStudio, IRIS, and Dr. Claw), marking each on whether it wraps a CLI agent, has built-in research state, in-place recovery, mid-run takeover, an end-user workspace, and unattended end-to-end operation. Marks were assessed from published papers, official documentation, and public repositories as of August 2026, not from runs.

Dr. Claw itself organizes work around four persistent state objects — Task Graph, Artifact Store, Decision Log, Execution Trace — layered over three system layers (Interaction, Orchestration, Execution) and iterating through Plan–Execute–Verify–Write-back. One interaction is modeled as a state transition, State(t+1) = f(State(t), Action(t), Obs(t)), and safety is modeled as Action(t) ∈ A(Policy(t)): actions are allowed only if they belong to the action space defined by the current permission configuration. Skills are directories with a SKILL.md manifest whose YAML frontmatter declares a name and description, commonly a version, license, allowed tools, and stage/domain tags; they reach tasks by stage-skill map resolution, keyword auto-load, or manual invocation. Backend adapters exist for the Claude and Codex SDKs (with Cursor hooks).

For evaluation, the researchers held the backend executor fixed — the codex provider with model gpt-5.4 under a matched danger-full-access/approval-never profile — so the only difference between bare codex and drclaw was the orchestration layer. Three medical problems were run with an identical, unenumerated instruction ("conduct a rigorous, publication-quality study…") and no rubric given to the agent: melanoma and nevus classification on Derm7pt, and a clinical-note risk baseline. Completeness was scored deterministically against the produced files as the fraction of 21 research best-practice elements spontaneously included, with no model-in-the-loop judgment. In the Dr. Claw condition the task graph was verified active (mean 17 tracked tasks).

For the human study, seven AI PhD participants were recruited, given a three-week free-use period to reduce unfamiliarity bias, and compared across three conditions (No-AI, Web/Desktop-AI, Dr. Claw) and three stages. Dr. Claw participants performed tasks with live logs, while the two controls were reported retrospectively from the most recent month. Statistics used Friedman tests with n = 7 matched participants and Holm-corrected pairwise Wilcoxon tests post-hoc.

Why This Matters

The paper shifts attention from "can an AI agent do research?" to "can a researcher stay in control of an AI that does research?" Its distinctive claim is architectural rather than model-level: the orchestration layer, not the underlying executor, determines whether a process is reviewable, recoverable, and auditable. Its ethics statement frames this directly — every stage passes through explicit human checkpoints, final acceptance rests with the researcher, and the Decision Log and Execution Trace keep a complete record of what was generated, approved, revised, or rejected, offered as a safeguard against flooding the literature with unverified output.

Real-world applications (as implied by the paper's design and evaluation):

• Academic and industrial research labs that need reproducible artifact trails for publications, internal reviews, or compliance.
• Medical and clinical AI research, the domain of the pilot tasks, where the authors require least-privilege permissions, retained operation traces, and human verification of citations, conclusions, and manuscript claims. They explicitly state outputs must not be used directly for real-world clinical decisions.
• Teams that already use command-line coding agents and want to add persistent state, approval gates, and multi-executor coordination without replacing their executor.
• Long-horizon or multi-person projects where a failed step must be recovered in place rather than restarting the pipeline.

Industry relevance: The paper targets the orchestration gap that tool vendors leave between chat, IDE, terminal, and writing environments. Its finding that agents skip hygiene elements (citations, limitations sections, subgroup analyses) unless a skill fires is directly relevant to anyone shipping agent products for regulated or publication-facing work, and its warning that suggestion is not invocation is a concrete reliability lesson for skill-library designers.

Future Directions

• Skill-only versus orchestration-only ablation. The authors call this "the natural next experiment," since the current comparison bundles task graph, state objects, skill library, and workflow instructions together.
• Verify skill execution rather than recommend it. Enforcing invocation instead of prompting a suggestion is named as the clearest reliability improvement the current design admits, and the one non-firing reference audit accounts for the single task where Dr. Claw failed to beat the bare agent.
• Expert per-artifact review. Completeness is a coverage measure, not a correctness check; establishing scientific soundness would require expert review of each produced artifact.
• Cross-domain transfer. All tasks come from a single domain (medical), so transferring the skill library and the structured Task Graph abstraction to other research areas — particularly open-ended work where rigid structure may add friction — remains unshown. The paper also notes that transfer to a new domain requires editing one JSON map rather than code.
• Larger, powered studies. The current evaluation uses a limited number of tasks and participants, and additional confounds remain, including prior familiarity with either interface and recall bias in the retrospective controls.

Target Audience

Researchers and engineers building AI-for-science systems, agent orchestration frameworks, or human-in-the-loop tooling will find the design-space comparison and the state-object abstraction most useful. Practitioners applying coding agents to open-ended, multi-stage work — especially in scientific or regulated settings where an audit trail matters — will benefit from the completeness breakdown showing where agents drop research hygiene. HCI researchers studying verification burden and context-switching costs will find the retrospective three-condition protocol and its stated validity threats relevant. Readers looking for new model capabilities or large-scale benchmarks will not find them here; the paper is explicit that its contribution is workflow integration, not model-level innovation.

Authors’ abstract

Command-line coding agents (e.g., Claude Code, Gemini CLI) can already read and write files and sustain long sessions, yet end-to-end research still fragments across chat tools, IDEs, terminals, and writing environments, and the decisions that make it auditable are rarely preserved. We present Dr. Claw, an open-source workspace that wraps existing coding-agent executors in a controllable and auditable human-in-the-loop workflow rather than introducing another autonomous agent. Persistent state objects, a reusable skill library, and multi-executor coordination link human decisions to AI execution, turning planning, execution, and writing into one traceable, recoverable loop. We demonstrate Dr. Claw through an interactive three-view scenario and a failure-recovery walkthrough, and evaluate it against a bare command-line agent sharing the same backend executor, so the comparison contrasts the whole orchestration layer (task graph, state objects, and skill library) with the agent it wraps. Holding the executor fixed, Dr. Claw scores higher on research completeness while persisting an auditable, recoverable process trail. Demo access: repository https://github.com/OpenLAIR/dr-claw, released under AGPL-3.0 with GPL-3.0 upstream components.