Temp-R1: A Unified Autonomous Agent for Complex Temporal KGQA via Reverse Curriculum Reinforcement Learning

AI summary

Overview

Research area: Natural Language Processing — Temporal Knowledge Graph Question Answering (TKGQA), LLM agents, and reinforcement learning for reasoning.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (GRPO, PPO-style objectives), Markov Decision Processes, retrieval-augmented generation, and knowledge graph embeddings.

One-sentence scope: The paper presents Temp-R1, an 8B-parameter autonomous agent trained with reinforcement learning and a hard-first curriculum that outperforms much larger closed-source pipelines on multi-hop temporal reasoning questions.

What This Paper Is About

Answering questions over temporal knowledge graphs — facts stored as (subject, predicate, object, timestamp) — is hard because correct answers often require chaining several facts together while respecting time constraints like "before," "first," or "in 2019." Existing systems solve this with fixed, hand-designed pipelines that call expensive closed-source APIs (GPT-4o, DeepSeek-V3), which limits flexibility and drives up cost. The authors build a single trainable agent that decides for itself when to search the graph, when to filter by temporal constraints, and when to rank results by date.

Key Contributions

1. An autonomous TKGQA agent with an expanded action space. Rather than lumping all reasoning into one <think> block, Temp-R1 decomposes internal cognition into three explicit, separately trainable actions — <plan>, <filter>, and <rank> — alongside the external <search> retrieval action and a terminal <answer>.
2. Reverse curriculum learning for reinforcement learning. Instead of the standard easy-to-hard progression, Temp-R1 trains on the hardest multi-hop questions first, then introduces simpler ones after a warm-up threshold. This prevents the policy from locking into shortcuts such as <search> → <answer> that work on easy samples but never activate complex tool chains.
3. A compact model that beats API-based pipelines. An 8B open-source backbone (Llama3.1-8B-Instruct) reaches state-of-the-art results on MultiTQ and TimelineKGQA, surpassing baselines built on GPT-4o and DeepSeek-V3, at zero inference-time API cost.
4. Empirical validation of every design choice. Ablations isolate the contribution of internal actions, the SFT cold start, and reverse curriculum learning, and training-dynamics plots show where each component stabilizes or destabilizes optimization.

Main Findings

• State-of-the-art overall performance at 8B scale: Temp-R1 scores 0.780 Hits@1 on the MultiTQ test set, edging past the strongest prior model (PoK, 0.779) while using an open-source backbone instead of GPT-4o-driven pipelines.
• Large gains on complex, multi-constraint questions: On the Multiple question category, Temp-R1 reaches 0.550 versus 0.459 for the best baseline (MemoTime), a 19.8% relative improvement. This is the paper's headline result and the clearest evidence that the agent learns genuine multi-hop temporal reasoning rather than surface patterns.
• High precision on time-typed answers: 0.969 Hits@1 on time answer types, indicating the model reliably pins down exact dates and intervals across granularities (day, month, year).
• Robust out-of-domain generalization: On the out-of-domain TimelineKGQA-ICEWS-Actor benchmark, Temp-R1 scores 0.642 overall versus 0.602 for PoK — while frozen GPT-4o collapses to 0.113, suggesting the trained policy transfers better than prompting-based systems under domain shift.
• Internal actions carry real weight: Removing <plan>/<filter>/<rank> drops overall accuracy from 0.780 to 0.620 and Multiple performance from 0.550 to 0.388. The authors attribute this to cognitive overload when all temporal reasoning must happen inside a single <think> tag.
• Reverse curriculum is the single largest ablation drop: Without it, overall accuracy falls to 0.556 and Multiple performance collapses to 0.143 — far worse than any other ablated configuration. Easy-first training plateaus early and depends on shortcuts.
• SFT cold start is a prerequisite for stable RL: Skipping it reduces overall accuracy to 0.582 and time-type accuracy from 0.969 to 0.713. Training curves show KL-loss spikes and early collapse without a supervised warm-up.
• Reasoning effort scales with difficulty: Complex questions trigger roughly twice as many <think> steps (1.36 → 2.93) and noticeably more <search> calls (1.33 → 1.92) than simple ones, showing the agent allocates computation adaptively rather than following a fixed sequence.
• Performance scales with backbone size and transfers across architectures: 7B beats 3B beats 1.5B on validation accuracy (0.790 vs. 0.532), and Llama and Qwen backbones perform comparably (0.780 vs. 0.790), indicating the method is not architecture-specific.

Methodology in Plain English

The authors recast question answering as a sequential decision problem. At each step, the model sees the original question plus everything that has happened so far, and chooses one action: write a plan, search the knowledge graph, filter retrieved facts against temporal constraints, rank the surviving facts by date, or emit a final answer. Only <search> produces new external information; the other internal actions are pure reasoning steps with no observation returned. This structuring exposes intermediate reasoning states to the learning algorithm, which is exactly what the <think>-only approach hides.

Training happens in three stages. First, a cold start: about 1,000 high-quality trajectories generated by GPT-4o and filtered for structural and answer correctness are used for supervised fine-tuning with a selective loss mask, so the model is only penalized on tokens it generates (tags and reasoning), not on retrieved content it copies. Second, reinforcement learning: the authors use Group Relative Policy Optimization (GRPO), sampling five trajectories per question, scoring each with a binary correct/incorrect reward, and normalizing rewards within the group to compute advantages. Third, the reverse curriculum: for the first portion of training only hard multi-hop questions are shown; simpler single-hop questions are added after a warm-up threshold.

The intuition behind hard-first ordering is that the model cannot succeed on difficult questions using trivial shortcuts, so it is forced to discover multi-step tool combinations. Once those are established, the simpler questions become easy cases the policy already handles.

Why This Matters

Impact on research. The paper challenges the prevailing TKGQA paradigm of chaining hand-designed prompt modules together with closed-source APIs. It shows that a small, locally trainable agent can match or exceed those systems, which shifts the research question from "how do we decompose the workflow" to "how do we design action spaces and curricula that let a policy discover its own workflow." The reverse-curriculum result is also relevant beyond temporal reasoning — any RL-trained agent trained on data with skewed difficulty distributions (which is most real data) faces the same shortcut trap.

Real-world applications:

• Financial and economic event analysis — answering questions like "Which company first announced layoffs after the merger?" over timestamped corporate event graphs.
• Geopolitical and news monitoring — the ICEWS benchmark underlying TimelineKGQA is built from political event data, directly relevant to tracking actor interactions over time.
• Clinical timeline reasoning — longitudinal patient records are inherently temporal and multi-hop (e.g., "Was drug X prescribed before or after the diagnosis?").
• Regulatory and compliance intelligence — auditors frequently ask "when" and "what changed first" questions over evolving policy or contract knowledge bases.

Industry relevance. The cost profile is the practical selling point. Prompt-based competitors pay closed-source API fees on every query; Temp-R1 performs fully local inference after training, and the paper reports roughly $50 in one-time cost for generating the SFT seed data. For organizations with high query volume over private temporal data, that changes the deployment calculus from "per-query API budgeting" to "one-time training plus commodity GPU inference."

Future Directions

• Scaling to larger backbones. The authors explicitly note they were limited to 8B parameters by compute. Whether Temp-R1's advantages hold, grow, or saturate at 14B and beyond is untested.
• Testing reverse curriculum outside temporal reasoning. The authors caution that their results do not establish the strategy as universal. Whether hard-first training helps or hurts in non-temporal, non-KG domains remains open.
• Richer reward signals. The current reward is a binary correct/incorrect signal at episode end. Process rewards, efficiency penalties, or penalties for unnecessary searches could reduce wasted tool calls and credit intermediate reasoning steps more finely.
• Adaptive curriculum scheduling. The warm-up threshold T₀ is set manually. Learning when to introduce easier examples — or scheduling based on measured policy competence — is an obvious refinement.

Target Audience

This paper benefits most readers working at the intersection of reinforcement learning and retrieval-augmented LLMs, particularly those building agents that must interleave reasoning with external tool calls. TKGQA and knowledge-graph researchers will find the benchmark comparisons and ablations directly actionable. Applied ML engineers evaluating whether a small self-hosted model can replace an expensive multi-agent API pipeline will find the cost analysis and efficiency tables the most immediately useful part. Readers without some RL background can still follow the core narrative — separate the reasoning steps, train hard first — but the GRPO formulation and training-dynamics analysis will require prior exposure.

Authors’ abstract

Temporal Knowledge Graph Question Answering (TKGQA) is inherently challenging, as it requires sophisticated reasoning over dynamic facts with multi-hop dependencies and complex temporal constraints. Existing methods rely on fixed workflows and expensive closed-source APIs, limiting flexibility and scalability. We propose Temp-R1, the first autonomous end-to-end agent for TKGQA trained through reinforcement learning. To address cognitive overload in single-action reasoning, we expand the action space with specialized internal actions alongside external action. To prevent shortcut learning on simple questions, we introduce reverse curriculum learning that trains on difficult questions first, forcing the development of sophisticated reasoning before transferring to easier cases. Our 8B-parameter Temp-R1 achieves state-of-the-art performance on MultiTQ and TimelineKGQA, improving 19.8% over strong baselines on complex questions. Our work establishes a new paradigm for autonomous temporal reasoning agents. The code is available at https://github.com/zjukg/Temp-R1.