Distilling Feedback into Memory-as-a-Tool

AI summary

Overview

Research area: Natural Language Processing — LLM agents, inference-time reasoning, memory-augmented generation, and learning from feedback.

Technical level: Intermediate. The core idea is intuitive, but the paper assumes familiarity with tool-calling agents, retrieval-augmented generation, self-critique pipelines, and rubric-based evaluation.

Scope: This paper proposes a file-based memory system that lets an LLM agent write generalizable "lessons learned" from evaluator feedback and retrieve them via tool calls, converting one-off critique into reusable knowledge.

What This Paper Is About

Modern LLMs can improve their answers by generating a draft, critiquing it, and revising — but this "System 2" style reasoning is expensive and ephemeral: the insight gained is thrown away the moment the context window closes, so the model re-derives the same corrections on every new task. This paper asks whether that transient feedback can instead be distilled into persistent, retrievable guidelines that the model reads and writes itself using file-based tools, so it can produce refined output zero-shot on future tasks at near-baseline cost.

Key Contributions

1. The Memory-as-a-Tool framework: A method for amortizing inference-time reasoning by having the LLM synthesize abstract principles from critique and write them to a persistent file system (/memories/) via explicit tool calls (ls, read_file, write_file, edit_file), then retrieve relevant notes before generating on subsequent tasks.

2. Distillation rather than logging: Memory does not store raw critiques or interaction history. It stores abstracted, structured, deduplicated rules — converting episodic feedback into semantic knowledge, including conflict resolution when new feedback contradicts old notes.

3. Rubric Feedback Bench: A new 42-scenario dataset for studying rubric-based learning, spanning five task categories: technical/visual writing analysis, chaotic/experimental "anti-rubric" writing, ethical reasoning under consequentialist vs. deontological constitutions, and AI behavioral/persona guidelines. Rubrics are multi-dimensional, weighted, and include behavioral descriptors across 4–5 performance levels.

4. Empirical validation across three frontier models: Continual-learning and long-horizon experiments on Claude Sonnet 4.5, GPT-5.1, and Gemini 3 Pro showing memory-augmented agents match or exceed self-critique performance while staying near baseline cost.

Main Findings

• Memory matches self-critique within two rounds: The Memory + Feedback approach starts at zero-shot baseline performance but rapidly improves, matching or exceeding the compute-heavy self-critique baseline after roughly two feedback rounds.

• Model-dependent learning speed: All three models benefited, but Claude Sonnet 4.5 showed the steepest learning curve, suggesting stronger ability to synthesize abstract rules from concrete feedback.

• Long-horizon robustness with mixed tasks: On 12 interleaved tasks of differing types, the memory agent scored 0.78 ± 0.10 versus 0.52 ± 0.25 for the no-feedback baseline — higher mean and markedly lower variance, indicating cross-domain transfer and reduced degradation.

• Memory consolidation is real: The agent accumulated 8 distinct memory files by the end of the long-horizon run, confirming it creates and maintains separate knowledge artifacts rather than a single blob.

• Cost–score Pareto improvement: The method achieves scores comparable to self-critique while keeping average cost per task close to the zero-shot baseline, because feedback cost is paid once during learning rather than on every query.

• Dramatic individual recoveries: Documented cases show scores jumping from 0/10 to 9.2/10 once the agent successfully retrieved and applied its learned guidelines.

• Interpretable memory by design: Because memory is human-readable text files with meaningful names (e.g., chaos_cinema_critique_rubric.txt), the agent's knowledge can be inspected, debugged, and manually edited — unlike opaque vector embeddings.

Methodology in Plain English

The pipeline has three phases per example. First, the model generates a response to a task prompt. Second, an evaluator LLM (a different Claude variant, held constant throughout experiments) reads the response against a private rubric and returns a critique plus a score. Third, the model decides whether to update its memory: it abstracts the specific complaint ("you failed to use synesthetic language in paragraph 2") into a general principle ("prioritize synesthetic blending"), then either writes a new file or edits an existing one, resolving contradictions with prior notes.

At test time, before generating, the agent calls ls to list memory filenames, reasons about which are relevant based on filename semantics and the current task, reads the chosen files, and prepends that content to its context. Retrieval is deliberate reasoning over filenames rather than embedding similarity search — simpler and more transparent, but dependent on the model maintaining descriptive filenames.

The framework was implemented two ways for fairness: the official Claude Code Python SDK (native filesystem tools, async streaming, Amazon Bedrock hosting) for Claude Sonnet 4.5, and a custom LangGraph/deepagents setup (InMemoryStore key-value backend, synchronous invocation, OpenRouter/Google AI) for GPT-5.1 and Gemini 3 Pro. Both used identical system prompts.

Evaluation compared three configurations: zero-shot base model with tools but no feedback; self-critique regenerating answers each step without persistence; and Memory + Feedback. The continual-learning protocol used horizons of 3 related tasks per rubric category across all 5 categories (15 examples per model), and the robustness protocol used 12 randomly interleaved mixed-category tasks on the best-performing model.

Why This Matters

Impact on research: The paper reframes inference-time reasoning from a per-query expense into a lifetime investment. Instead of a binary choice between costly test-time search and costly parameter fine-tuning, it offers a third path — non-parametric, continual, and inspectable. It also connects self-critique literature (Self-Refine, Reflexion), rubric-feedback work, and agentic memory architectures into a single pipeline, and offers a ready benchmark for others to build on.

Real-world applications:

• Content and copywriting assistants that learn a client's house style and voice from editorial feedback, then apply it automatically on future briefs without re-reviewing every draft.
• Safety and compliance agents that accumulate behavioral guidelines from incident feedback, so policy constraints persist across sessions rather than being restated in every system prompt.
• Domain-specific drafting tools (legal, medical, technical writing) where expert reviewers give rubric-based critique and the system compounds those corrections over time.
• Customer-facing assistants that internalize escalation, tone, and persona rules from human quality-assurance reviews.

Industry relevance: The cost-efficiency result is the commercially salient finding. Self-critique pipelines roughly double or triple token usage per query; this approach pays that premium once and then runs near-zero-shot economics. For any product serving high query volume, that is a direct reduction in serving cost with no apparent quality penalty — and because memory is plain text, it fits naturally into existing observability and audit workflows.

Future Directions

• Scaling retrieval beyond a handful of files. The current mechanism relies on the LLM reading filenames, which will degrade as memory grows to thousands of entries. The authors propose hierarchical directory structures with recursive ls traversal, mimicking how human experts navigate large repositories, plus hybrid retrieval combining filename reasoning with semantic search.

• Active forgetting and memory pruning. Over lifetimes of thousands of episodes, stale, redundant, or contradictory memories could clutter context. Mechanisms for deliberate forgetting and knowledge decay are not yet addressed.

• Systematic study of memory content quality. The paper shows memory files are high-signal, but does not quantify how much abstraction quality drives performance, or what happens when the agent abstracts incorrectly and writes a harmful rule.

• Beyond rubric-based writing tasks. All evaluation scenarios are open-ended writing and behavioral/ethical reasoning. Whether the approach transfers to code generation, tool-use planning, or multi-agent settings remains open.

• Interaction with fine-tuning. The paper positions memory as an alternative to parameter updates; combining the two — using distilled memory as training data for lightweight adaptation — is an unexplored hybrid.

Target Audience

This paper is most valuable to LLM agent researchers and engineers building persistent-memory or self-improvement systems, to practitioners who need to cut inference costs on high-volume generation products without losing output quality, and to teams working on alignment and specification-following who want an auditable, editable store of behavioral rules rather than opaque weights. Evaluation researchers and dataset builders will also find the Rubric Feedback Bench useful as a ready-made testbed for continual learning from structured feedback.

Authors’ abstract

We propose a framework that amortizes the cost of inference-time reasoning by converting transient critiques into retrievable guidelines, through a file-based memory system and agent-controlled tool calls. We evaluate this method on the Rubric Feedback Bench, a novel dataset for rubric-based learning. Experiments demonstrate that our augmented LLMs rapidly match the performance of test-time refinement pipelines while drastically reducing inference cost.