EffGen: Enabling Small Language Models as Capable Autonomous Agents

AI summary

Overview

• Research area: Natural language processing, specifically autonomous language model agents and small language model (SLM) deployment.
• Technical level: Intermediate. The paper includes formal notation (complexity scoring functions, dependency graphs) but the core ideas are accessible to anyone familiar with LLM agent frameworks.
• Scope: The paper introduces EffGen, an open-source agentic framework that redesigns standard agent components—prompting, routing, decomposition, memory, and protocol support—specifically around the constraints of small language models (up to ~32B parameters), and evaluates it against established frameworks across 13 benchmarks.

What This Paper Is About

Most agent frameworks today (LangChain, AutoGen, and similar tools) are built assuming access to large proprietary models like GPT, Claude, or Gemini, which makes them expensive to run and unsuitable for privacy-sensitive tasks. The problem is that small language models—those small enough to run locally on a laptop or a single GPU—struggle with the complex prompts, long contexts, and multi-step reasoning these frameworks demand. EffGen's goal is to close that gap by redesigning each piece of the agent pipeline with small-model constraints treated as first-class requirements rather than afterthoughts.

Key Contributions

1. SLM-aware prompt optimization. A rule-based pipeline that compresses prompts by 57% on average (up to 70–80%) through phrase simplification, sentence splitting, redundancy removal, bullet formatting, and context-aware truncation, while preserving task semantics.

2. Pre-execution complexity routing. A five-factor scoring function (task length, requirement count, domain breadth, tool requirements, reasoning depth) that predicts task complexity before any inference occurs, allowing simple queries to run on a single agent and complex ones to be routed to parallel, sequential, hybrid, or hierarchical execution.

3. Intelligent task decomposition with dependency graphs. A system that breaks complex queries into subtasks annotated with specializations (research, coding, analysis, etc.) and orders them via parallel, sequential, hybrid, or hierarchical strategies, using the same local model rather than calling out to an external API.

4. Unified multi-protocol support and three-tier memory. A single implementation of MCP, A2A, and ACP protocols for cross-ecosystem interoperability, plus a memory system combining short-term conversation history, long-term episodic storage with importance scoring, and vector-based semantic retrieval.

Main Findings

• Smaller models benefit far more from framework design. EffGen improves Qwen2.5-1.5B by 13.2 percentage points over the next-best baseline, but only 6.0 points at 32B—suggesting raw model capability partially compensates for framework inefficiency at larger scales.

• Prompt optimization and routing have complementary scaling behavior. Prompt optimization helps small models more (11.2% gain at 1.5B vs. 2.4% at 32B), while complexity routing helps large models more (3.6% at 1.5B vs. 7.9% at 32B). Combined, they deliver consistent gains at every scale.

• Code execution benefits 26× more than calculator-style tasks. On BeyondBench-Hard (NP-complete problems), EffGen lifts Qwen2.5-7B from 15.42% to 28.57%, with cross-framework variance of 12.4% versus just 4.2% for math tasks.

• Tool schema quality, not model capability, drives multi-tool performance. On SimpleQA, EffGen reaches 40%/76%/84% at 1.5B/7B/32B, versus Smolagents at 18%/32%/52%. A control baseline with direct tool access but no scaffolding (Raw+Search) reached only 8% at 1.5B, confirming the scaffolding—not tool availability—is the differentiator.

• Execution speed scales near-linearly. EffGen achieves up to 18× speedup at 1.5B parameters versus Smolagents, while LangChain shows super-linear slowdowns (11.2× time increase for 21.3× parameter growth).

• Pre-execution routing cuts wasted computation by 23% compared to frameworks that discover complexity during execution.

• Agentic frameworks help selectively, not universally. On GSM8K, tool-calling overhead actually hurts LangChain at small scales; EffGen's prompt optimization alone lifts the 1.5B raw model from 68.01% to 71.63%.

Methodology in Plain English

The researchers built EffGen from the ground up around the idea that small models need different treatment than large ones. The system first scores each incoming query on a 0–10 complexity scale using five weighted signals (how long the question is, how many requirements it contains, how many knowledge domains it touches, how many tool types it seems to need, and how deep the required reasoning appears to be). Based on that score against a default threshold of 7.0, the query either runs through a simple ReAct-style single-agent loop or gets decomposed into subtasks arranged in a dependency graph.

Before any of this, prompts are run through a model-size-aware optimizer—Tiny models (<1B) get more aggressive compression than Medium ones (3–7B)—that strips politeness phrases, splits long sentences at conjunctions, and converts instructions to bullets. A three-tier memory system tracks the conversation, stores important episodes with recency/frequency/importance scoring, and retrieves semantically similar past content via embeddings.

The team evaluated EffGen on 13 benchmarks spanning math, code execution, retrieval-augmented reasoning, memory, and multi-tool agentic tasks, using Qwen2.5 (1.5B–32B), Gemma 3, and GPT-OSS 20B, comparing against raw model prompting, LangChain, AutoGen, and Smolagents under identical tool sets, timeouts, and generation parameters. Statistical significance was verified with McNemar's test.

Why This Matters

Impact on research. The paper challenges the assumption that agentic capability scales primarily with model size, and its finding that prompt optimization and routing have complementary scaling behavior offers a concrete direction for future framework design. It also provides a rare controlled result showing tool access alone (Raw+Search) isn't sufficient—scaffolding is.

Real-world applications:

• Privacy-sensitive deployments. Healthcare, legal, and financial use cases where data cannot leave the local environment can now run capable agents entirely on local hardware.
• Cost-constrained production systems. Startups and research labs can deploy agents without paying per-token API costs, with EffGen shipping production infrastructure (guardrails, observability, rate limiting, Docker/Helm/Lambda deployment).
• Edge and offline environments. Field operations, embedded systems, or air-gapped networks where cloud connectivity is unavailable.
• Interoperable agent ecosystems. The unified MCP/A2A/ACP implementation lets EffGen agents plug into broader agent networks regardless of protocol.

Industry relevance. The framework's Apache 2.0 license, PyPI package (pip install effgen), and 66 built-in tools lower the barrier to adoption. The explicit conflict-of-interest disclosure—one author works at Google DeepMind, which develops Gemma 3—signals the paper's engagement with real commercial model ecosystems.

Future Directions

• Extending beyond the 32B ceiling. The paper's own data shows improvements shrink as model size grows; whether EffGen's optimizations remain useful (or even measurable) at 70B+ or frontier scales is untested.

• Adapting routing thresholds automatically. The default thresholds (τ=7.0, τ_H=8.5, τ_hier=9.0) were empirically tuned; whether per-task or per-model calibration could improve results further is unexplored.

• Handling newer reasoning models. The evaluation focuses on standard instruction-tuned models; how EffGen's routing and prompt compression interact with chain-of-thought-native models (like reasoning-tuned variants) is an open question.

• Improving decomposition quality. The paper notes that all decomposition uses the same small model rather than an external LLM—this keeps costs down but may limit subtask quality on very complex queries. Whether a hybrid approach helps is untested.

Target Audience

This paper is most valuable for ML engineers and applied researchers building agent systems under compute or privacy constraints, particularly those working with models in the 1B–32B range. It's also useful for framework developers interested in how prompt optimization and routing interact, and for product teams at startups or enterprises evaluating whether local SLM agents can replace API-based LLM pipelines. Readers should have basic familiarity with LLM agents, tool-calling, and concepts like ReAct and RAG to get the most from it.

Authors’ abstract

Most existing language model agentic systems today are built and optimized for large language models (e.g., GPT, Claude, Gemini) via API calls; while powerful, this approach faces several limitations including high token costs and privacy concerns for sensitive applications. We introduce EffGen, an open-source agentic framework optimized for small language models (SLMs) that enables effective, efficient, and secure local deployment. EffGen makes four major contributions: (1) Enhanced tool-calling with prompt optimization that compresses input prompts by up to 70-80% (and 57% on average across our benchmarks) while preserving task semantics, (2) Intelligent task decomposition that breaks complex queries into parallel or sequential subtasks based on dependencies, (3) Complexity-based routing using five factors to make smart pre-execution decisions, and (4) Unified memory system combining short-term, long-term, and vector-based storage. Additionally, EffGen unifies multiple agent protocols (MCP, A2A, ACP) for cross-protocol communication. Results on 13 benchmarks show EffGen outperforms LangChain, AutoGen, and Smolagents with higher success rates, faster execution, and lower memory. Our results reveal that prompt optimization and complexity routing have complementary scaling behavior: optimization benefits SLMs more (11.2% gain at 1.5B vs 2.4% at 32B), while routing benefits large models more (3.6% at 1.5B vs 7.9% at 32B), providing consistent gains across all scales when combined. EffGen is released under the Apache 2.0 License, ensuring broad accessibility for research and commercial use, with the code available at https://github.com/ctrl-gaurav/effGen, the Python package at https://pypi.org/project/effgen/ (pip install effgen), and the project website and documentation at https://effgen.org/ and https://docs.effgen.org/.