TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents

AI summary

Overview

Research area: Large language model (LLM) research agents and automated scientific discovery workflows.

Technical level: Intermediate. The paper assumes familiarity with agentic workflows, tool calling, and LLM prompting, but the system itself is presented as a lightweight, low-configuration tool.

Scope: The paper introduces TinyScientist, a modular framework for building research agents that decomposes automatic research into four stages (think, code, write, review) and adds feature components for interactivity, extensibility, and controllability, released as an open-source codebase, a web demo, and a PyPI package.

What This Paper Is About

Existing automatic research systems built on LLMs tend to be complex and hard to use, extend, or control without significant technical expertise. The authors identify three specific problems: weak interactivity between human researchers and agents, rigid tool-specific designs that resist extension, and insufficient supervision over safety, ethics, and cost budgets. TinyScientist addresses these by providing a lightweight, hierarchical framework whose core workflow stays fixed while tools, interfaces, and safety/budget controls can be swapped or adjusted.

Key Contributions

1. A framework design organized around four modular workflow components (Thinker, Coder, Writer, Reviewer) governed by a top-level Engine class, with each component supported by reusable feature components (InputFormatter, OutputFormatter, MCPClient, Checker).
2. A tabular-based communication interface that decomposes the research workflow into editable stages, letting researchers modify individual cells or columns (for example, adding new baselines as rows) rather than issuing open-ended instructions.
3. Adoption of the Model Context Protocol (MCP) instead of direct API wrapping, giving a unified way to connect tools such as code searchers, plot drawers, and paper retrievers to each workflow stage.
4. A Checker component enforcing budget and safety constraints, plus three released artifacts: an open-source codebase at https://github.com/ulab-uiuc/tiny-scientist, an interactive web demonstration at https://app.auto-research.dev, and a PyPI package at https://pypi.org/project/tiny-scientist.

Main Findings

• Generation quality versus Agent Laboratory: Papers generated by TinyScientist had writing quality and idea quality that were better, or at least comparable in all cases, than those produced by Agent Laboratory, a widely used multi-agent auto-research framework, under both LLM-based and human evaluation. The largest improvement appeared in the biological writing domain, with a 0.23-point increase under both evaluation settings.
• Out-of-distribution performance drops: For out-of-distribution biological cases, both TinyScientist and Agent Laboratory showed lower performance, which the authors attribute to prompt design being primarily tailored to the ML domain.
• Tool usage helps: In an ablation on ML-related tasks, augmenting paper generation with tools such as the paper searcher produced more than a 0.1-point gain in writing quality and around a 0.05-point gain in idea quality. Reported scores: with tool, LLM evaluation writing 4.06 and idea 3.82, human evaluation writing 3.93 and idea 3.83; without tool, LLM evaluation writing 3.93 and idea 3.78, human evaluation writing 3.80 and idea 3.68.
• Safety checker blocks unsafe intents: Of 20 biology-related potentially unsafe tasks drawn from SciSafetyBench, 18 were blocked at the thinker stage and the remaining 2 were flagged with warnings at further workflow stages.
• Budget checker adapts the workflow: The checker estimates cost and dynamically adjusts the number of reflection steps within each stage (thinker, writer, coder, reviewer, checker) based on the budget and backbone model. If actual cost exceeds the budget due to estimation errors, the checker terminates the process early to prevent overspending.
• Usability: A complete research paper PDF in standard AI conference format, along with the research idea, experimental code, and peer review details, can be generated with seven lines of code using the Python package.

Methodology in Plain English

The authors first break an automated research project into four sequential stages. The Thinker samples a number of initial ideas from a user-supplied intent and refines each through several rounds, producing a description, an experimental plan, a comparison table with related work, and self-evaluation scores for impact, feasibility, and novelty. The Coder takes an idea and its plan and drives an external coding agent framework such as Aider to produce and run code, pausing for human input if execution fails or deviates from the plan. The Writer produces drafts in three steps (initial generation, refinement against an error checklist for LaTeX issues, then citation insertion using retrieval through the Semantic Scholar API). The Reviewer simulates peer reviews containing a summary, strengths, and weaknesses, refines each through self-reflection, and synthesizes a meta-review with a final score.

Around these four stages sit the feature components. The Formatter compiles stage outputs into structured tables that humans can read and edit. The MCPClient connects external tools to the stages. The Checker enforces cost and safety limits, blocking harmful outputs at stage-specific filters and adjusting iteration counts to stay within budget.

The evaluation compares TinyScientist against Agent Laboratory using gpt-4o-mini as the backbone model for both. Inputs are 20 machine-learning-related ideas randomly sampled from Si et al. (2024), using idea titles as intents, plus 20 biology-related potentially unsafe tasks from SciSafetyBench. Quality is judged on writing quality (clarity, appropriate citation use, richness) and idea quality (research value and usefulness) using the same rubric for a GPT-4o LLM judge and for human annotators with relevant research backgrounds.

Why This Matters

Impact on research: The paper positions TinyScientist as a way to lower the barrier for researchers and developers entering automatic research, arguing that simpler, more modular, and more controllable agent frameworks are easier to extend and maintain as algorithms and architectures advance. Notably, TinyScientist is described in the comparison table as uniquely combining tabular-based human-agent communication, schematic diagram design, MCP-based tool calling, and budget/safety control within a deployment-ready system.

Real-world applications:

• Generating draft research papers, experimental code, and reviews from a plain-text intent.
• Domain-specific research assistance where tools such as paper searchers and plot drawers are swapped in through MCP.
• Safety screening of research intents, as demonstrated by blocking 18 of 20 potentially unsafe biology tasks.
• Cost-governed deployment, where stage-level budgets constrain the number of reflection steps and terminate runs early if spending exceeds the estimate.

Industry relevance: The framework is released as a PyPI package and a hosted web demonstration, and the authors state that generated outputs are watermarked to indicate AI involvement and are not intended for direct use in academic publishing without human verification.

Future Directions

• Improving compilation stability by adding better context sanitization and format-control generation, since LaTeX formatting inconsistencies still cause failures such as missing references or line misalignment.
• Making LaTeX error and warning correction a more robust part of the output formatter in future versions of TinyScientist.
• Raising the visual quality, consistency, and content-grounding of generated diagrams for sections such as the Introduction and Method.
• Broadening prompt design beyond the ML domain, since both evaluated frameworks performed worse on out-of-distribution biological cases.

Target Audience

Researchers and developers who want to build, extend, or study automated research agents without deep infrastructure work, along with practitioners interested in human-in-the-loop scientific workflows and in safety and cost controls for agentic systems.

Authors’ abstract

Automatic research with Large Language Models (LLMs) is rapidly gaining importance, driving the development of increasingly complex workflows involving multi-agent systems, planning, tool usage, code execution, and human-agent interaction to accelerate research processes. However, as more researchers and developers begin to use and build upon these tools and platforms, the complexity and difficulty of extending and maintaining such agentic workflows have become a significant challenge, particularly as algorithms and architectures continue to advance. To address this growing complexity, TinyScientist identifies the essential components of the automatic research workflow and proposes an interactive, extensible, and controllable framework that easily adapts to new tools and supports iterative growth. We provide an open-source codebase, an interactive web demonstration, and a PyPI Python package to make state-of-the-art auto-research pipelines broadly accessible to every researcher and developer.