ForeDreamer: A Self-Evolving Dual-Agent Memory Architecture for Future Event Prediction

AI summary

Overview

• Research area: Natural Language Processing — LLM agent memory, web-augmented retrieval, and automated future event prediction (forecasting).
• Technical level: Advanced. The paper combines a two-agent orchestration design, a formal task formulation with temporal cutoffs, tree-structured procedural memory, and validation-gated self-evolution.
• Scope (one sentence): The paper introduces ForeDreamer, a self-evolving dual-agent memory architecture that converts noisy open-web search results into question-specific "factual memory" and evolves two forms of persistent "experiential memory" across forecasting episodes, evaluated on Prophet Arena and FutureX.

What This Paper Is About

Agents that predict future real-world events must find evidence on the open web and turn messy, redundant, time-sensitive, and sometimes conflicting reports into a usable basis for a calibrated forecast. Existing approaches largely either pass retrieved text straight into the model context, or use memory systems built for cleaner, user-centered dialogue tasks that mainly store, retrieve, and reuse prior information.

The goal of ForeDreamer is to insert a structured evidence-processing step between retrieval and prediction, and then to let the agent improve that step and its forecasting behavior over time using feedback from past forecasting episodes.

Key Contributions

1. Formalizing factual-memory management for open-ended forecasting (RQ1). The paper defines factual memory as a query-conditioned evidence artifact for the current forecasting question and proposes ForeDreamer, which separates forecasting from evidence processing: a main agent searches, integrates evidence, and predicts, while a memory-processing subagent builds factual memory under a MemGuide (workflow specification) and executable MemTools (operations).
2. A dual-track self-evolution mechanism for experiential memory (RQ2). One track evolves textual forecasting experience in an Experience Bank that guides search planning, evidence integration, and forecast calibration; the other track evolves procedural evidence-processing experience — the MemGuide tree and MemTools — that determines how search results become factual memory.
3. Two optimizations for procedural evolution. After analyzing the vanilla evolution process, the authors identify redundant tool generation and over-concentrated exploration, and introduce compositional tool reuse and diversity-guided exploration.
4. Empirical validation. Experiments on Prophet Arena and FutureX with Qwen3.5-Flash and GPT-5.4-Nano, including dual-track ablations, optimization ablations, robustness across search settings, a data-leakage check, and analyses of tool similarity and MemGuide pipeline archetypes.

Main Findings

• Prophet Arena, Qwen3.5-Flash (average Brier score, lower is better): ForeDreamer reaches 0.1471, compared with Full Text 0.2059, RAG 0.1876, HippoRAG 2 0.1831, Mem0 0.2110, MemoryOS 0.1771, A-MEM 0.1862, LightMem 0.1761, and LangMem 0.1868.
• Prophet Arena, GPT-5.4-Nano (average Brier score, lower is better): ForeDreamer reaches 0.1839, compared with Full Text 0.2084, RAG 0.1997, HippoRAG 2 0.2209, Mem0 0.2307, MemoryOS 0.2073, A-MEM 0.2186, LightMem 0.2123, and LangMem 0.2146.
• FutureX, Qwen3.5-Flash (accuracy, higher is better): ForeDreamer reaches 0.4108, compared with Full Text 0.3298, RAG 0.3382, HippoRAG 2 0.3285, Mem0 0.3264, MemoryOS 0.3269, A-MEM 0.3495, LightMem 0.2885, and LangMem 0.3333.
• FutureX, GPT-5.4-Nano (accuracy, higher is better): ForeDreamer reaches 0.3883, compared with Full Text 0.2766, RAG 0.3221, HippoRAG 2 0.3173, Mem0 0.3567, MemoryOS 0.3262, A-MEM 0.3480, LightMem 0.2788, and LangMem 0.3333.
• Both evolution tracks matter (Qwen3.5-Flash): Full Text scores 0.2059 on Prophet Arena and 0.3298 on FutureX; removing MemGuide and MemTool evolution gives 0.1663 and 0.3892; removing Experience Bank evolution gives 0.1769 and 0.3351; the full system gives 0.1471 and 0.4108.
• Both procedural optimizations matter (Qwen3.5-Flash): removing both gives 0.1592 on Prophet Arena and 0.3850 on FutureX; removing compositional tool reuse gives 0.1541 and 0.4032; removing diversity-guided exploration gives 0.1554 and 0.3564; the full system gives 0.1471 and 0.4108.
• Redundant tools in vanilla evolution: on FutureX, 201 generated MemTools collapse into 98 clusters at a token-Jaccard threshold of τ = 0.5, a cluster ratio of 48.8%.
• Concentrated exploration in vanilla evolution: all 10 FutureX MemGuides evolved with vanilla expansion fall into the same tool-orchestrated pipeline archetype.
• Robustness across search settings: ForeDreamer improves over Full Text across alternative configurations, including maximum interaction turns of 3 and 4, Firecrawl search, Top-2, Top-6, and Top-8 retrieved results, and 60K and 90K search contexts. Some alternative settings shift the numbers — for example, on FutureX with Firecrawl search, ForeDreamer scores 0.4596 and Full Text 0.4394, and with a maximum of 3 interaction turns ForeDreamer scores 0.4423 and Full Text 0.3894. On Prophet Arena, Top-6 retrieval gives ForeDreamer 0.1426 versus the 0.1471 base setting.
• Data-leakage check: without web search (No Information), Qwen3.5-Flash scores 0.2545 on Prophet Arena and 0.3077 on FutureX; GPT-5.4-Nano scores 0.2492 and 0.2260. The gap to the search-based settings is presented as evidence that external web evidence, not the backbone model alone, drives the gains.
• Benchmark scale: Prophet Arena is used as a resolved snapshot of 1200 forecasting questions across eight categories (Climate/Weather, Companies, Economics, Entertainment, Mentions, Other, Politics, Sports), with snapshot times from June 17, 2025 to November 16, 2025, and a 400-example subset used for Sports. FutureX is a resolved split of 208 questions with end times from January 15, 2026 to March 10, 2026.

Methodology in Plain English

The system treats each forecasting question as a pair of the question plus a temporal cutoff that defines which information would have been available at prediction time. Everything is constrained to evidence available at or before that cutoff.

Two agents share the work. The main agent holds the question, consults an Experience Bank of reusable textual forecasting lessons, plans search queries, and runs a multi-turn search-and-reasoning loop with a turn budget. The memory-processing subagent receives each retrieved item and turns it into concise, question-specific factual memory. It does this by following a MemGuide, which describes the processing workflow, and calling MemTools, which are executable operations such as reading source evidence, extracting salient information, creating intermediate representations, validating content, and writing the final factual memory.

Processed results are returned as evidence, not as instructions that can change the main agent's control flow, and the main agent only sees a single public search-and-process interface — the internal retrieval and processing machinery stays hidden. On the final allowed turn, the agent answers without calling the search tool and returns a forecast plus a rationale. Each rollout is recorded with interactions, artifacts, ground truth, and metrics, and becomes feedback for evolution.

Evolution happens on a small evolution-and-validation pool and runs on two tracks. In the textual track, an editor proposes one of three operations on the Experience Bank — Add, Modify, or Remove — and the edited bank is kept only if it improves validation performance. In the procedural track, procedural memory is a tree of MemGuides with edges recording provenance; parent guides are chosen by validation-rank Zipf sampling, so stronger guides are favored while lower-ranked ones still have nonzero probability. A critic derives a design requirement from a rollout, a candidate MemGuide and MemTools are generated, and the candidate is admitted only if it passes interface checks and validation on all pool examples.

Two refinements address vanilla evolution's weaknesses. Compositional tool reuse first selects a compatible subset of existing valid MemTools for the design requirement and generates new tools only for operations not already covered, cutting near-duplicate tool creation. Diversity-guided exploration adds a scheduler-driven path that generates category-diverse candidates from MemGuide-category summaries and representative guides without needing a new rollout, so evolution is not limited to refining one locally successful guide family.

Evaluation uses Qwen3.5-Flash and GPT-5.4-Nano as backbones, temporally constrained web search for evidence, Brier score on Prophet Arena and accuracy on FutureX, and baselines of Full Text, a generic RAG baseline, HippoRAG 2, Mem0, MemoryOS, A-MEM, LightMem, and LangMem.

Why This Matters

Impact on research. The paper argues that memory for evidence-heavy open-world tasks should be treated as an evolving processing capability rather than a storage-and-retrieval utility. It draws a distinction between question-specific factual memory and cross-episode experiential memory, and shows that evolving both — including the executable tools used to process evidence — outperforms both raw retrieval and existing agent-memory systems on the two benchmarks used.

Real-world applications (from the forecasting setting the paper targets):

• Decision support for prediction markets and event contracts, where calibrated probabilities matter more than a single yes/no answer.
• Monitoring and risk analysis for time-sensitive domains such as politics, economics, companies, and climate/weather, where the relevant signal is buried in noisy reporting.
• Automated research assistance that must assemble an auditable evidence trail, since each piece of factual memory can be traced back to the retrieved item it came from.
• Tool-using agent pipelines that need to convert messy retrieved content into structured, task-specific inputs before a downstream model acts on them.

Industry relevance. The work comes from Zhejiang University's State Lab of CAD & CG, Ant Group, the National University of Singapore, and Zhejiang University of Technology, and its design choices — bounded tool execution budgets, isolated execution environments, allowlisted tools, and a validation gate before any evolved asset is published — speak directly to concerns about deploying self-modifying agents in production. The paper does not report latency, cost, or deployment figures, so those practical trade-offs are not quantified here.

Future Directions

• Reducing reliance on a small feedback pool. The limitations section notes that resolved forecasting feedback is scarce, so evolution and validation use a small pool. The authors argue that held-out questions show the memory generalizes rather than fitting validation items, but how to get more resolved feedback remains open.
• Extending beyond forecasting. The authors explicitly place traditional agent-memory benchmarks, which test storing, retrieving, and reusing clean interaction histories, outside the scope of this work. Whether the factual/experiential separation transfers to those settings is untested here.
• Broadening robustness evidence. The robustness table varies interaction turns, search provider, number of retrieved results, and search context size, with some settings producing better ForeDreamer numbers than the base configuration. Understanding which configuration is genuinely best, and why, is left open. The paper notes that additional results with GPT-5-Nano, whose training-data cutoff precedes the evaluated benchmark periods, appear in an appendix, and further no-information checks are also deferred to an appendix.
• Going beyond single-stage procedural evolution. The paper's own analysis shows vanilla evolution collapses into one pipeline archetype; diversity-guided exploration mitigates this, but how much archetype diversity is optimal, and how to manage it automatically, is not resolved.

Target Audience

This paper is most useful to researchers and engineers working on LLM agents, agent memory architectures, retrieval-augmented generation for noisy web data, self-improving or self-evolving agent systems, and automated forecasting or prediction-market applications. It also suits practitioners who need to build agent pipelines where retrieved evidence must be structured and auditable before a downstream decision, and readers familiar with the baseline memory systems it compares against (HippoRAG 2, Mem0, MemoryOS, A-MEM, LightMem, LangMem).

Authors’ abstract

Open-web future event prediction requires agents to distill reliable signals from noisy, redundant, and incomplete evidence. Existing retrieval/memory mechanisms directly feed retrieved information to agents or rely on simple memory functions such as storing and reusing prior information for prediction, leaving them insufficient for open-web forecasting. We propose to transform raw web evidence into structured memory before prediction, enabling agents to reason over distilled, question-specific evidence rather than noisy retrieval results. This paper presents ForeDreamer, a self-evolving dual-agent framework for managing memory over open-web evidence. ForeDreamer separates factual memory, a question-specific evidence state for the current forecast, from experiential memory, persistent agent experience accumulated across forecasting episodes. It uses a main agent for search and prediction, and a memory-processing subagent to convert search results into factual memory with dedicated tools. ForeDreamer further evolves experiential memory through two tracks, improving both forecasting decisions and factual-memory construction. Experiments on Prophet Arena and FutureX demonstrate the effectiveness of ForeDreamer. Project page: https://zhongzero.github.io/ForeDreamer