Trajectory Guard -- A Lightweight, Sequence-Aware Model for Real-Time Anomaly Detection in Agentic AI

AI summary

Overview

Research area: Anomaly detection for LLM-based autonomous agent trajectories, positioned at the intersection of sequential anomaly detection and agentic AI safety.

Technical level: Intermediate. The paper assumes familiarity with embedding models, autoencoders, contrastive learning, and recurrent networks, but the architecture and motivation are explained without heavy mathematical formalism.

Scope: The paper proposes and evaluates Trajectory Guard, a Siamese Recurrent Autoencoder with a hybrid contrastive-plus-reconstruction loss, for real-time detection of anomalous multi-step LLM agent plans.

What This Paper Is About

LLM-based autonomous agents produce multi-step action plans ("trajectories"), and those plans can fail either because they do not fit the task at hand or because their internal step sequence is malformed. The paper argues that existing anomaly detection approaches are unsuitable for this problem: mean-pooling embeddings dilutes anomalous steps, and contrastive-only methods ignore sequential structure, with standard unsupervised methods on pre-trained embeddings reaching F1-scores no higher than 0.69. The goal is a lightweight model that detects both failure modes in one pass, fast enough for real-time safety verification.

Key Contributions

1. Demonstrates empirically that standard anomaly detection methods applied to pre-trained embeddings (VAE, Isolation Forest, One-Class SVM) are ineffective for agent trajectory validation, establishing the need for specialized models.
2. Proposes a sequence-aware Siamese Recurrent Autoencoder with a hybrid loss combining contrastive task-trajectory alignment and recurrent reconstruction for real-time trajectory anomaly detection.
3. Evaluates on benchmarks combining synthesized data from Galileo and AgentAlign with real-world failures from RAS-Eval and Who&When, reporting F1-scores of 0.88–0.94 on synthetic benchmarks and recall of 0.86–0.92 on real-world logs.
4. Demonstrates inference more than 17× faster than LLM Judge baselines, and contributes a deployable tool for verifying trajectory coherence against tasks in agentic systems.

Main Findings

• Standard detectors are inadequate: Unsupervised methods on pre-trained embeddings (VAE with bge-large, One-Class SVM, Isolation Forest) achieve F1-scores no higher than 0.69 on this task, with the VAE at F1 0.69, One-Class SVM at 0.55, and Isolation Forest at 0.53 on mixed synthetic data. The authors attribute this to mean-pooling diluting the anomalous steps.

• Contextual contrastive fine-tuning helps but is brittle: Fine-tuning all-MiniLM-L6-v2 contrastively on (task, trajectory) pairs with MultipleNegativesRankingLoss improved performance to F1 ≈ 0.82, but the authors report negative transfer across trajectory formats and note that sequential structure is ignored.

• The hybrid loss is the decisive component: An ablation study on a balanced validation set shows contrastive loss alone yields anomaly F1 0.82, reconstruction loss alone yields 0.75, and the hybrid loss yields 0.92 — the full model outperforms either component in isolation.

• Strong results on balanced synthetic benchmarks: Trajectory Guard scores F1 0.88 on Galileo (precision 0.90, recall 0.86), F1 0.94 on AgentAlign (precision 0.95, recall 0.93), and F1 0.92 on the mixed synthetic weighted average (precision 0.94, recall 0.91).

• Competitive with heavyweight LLM judges on synthetic data: On mixed synthetic data, Gemini Flash 1.5 reports F1 0.95 (P 0.93, R 0.98), GPT-4o Mini F1 0.92 (P 0.89, R 0.97), Deepseek v3.2-exp F1 0.85 (P 0.77, R 0.94), and Phi-3-mini F1 0.81 (P 0.80, R 0.82). Trajectory Guard's F1 0.92 matches GPT-4o Mini and exceeds Deepseek and Phi-3-mini.

• Recall on real-world hold-outs, where false negatives matter most: On the imbalanced external benchmarks the paper reports recall only, due to lack of normal samples. Trajectory Guard achieves recall 0.86 on RAS-Eval and 0.92 on Who/When, versus Phi-3-mini at 0.76 and 0.88, GPT-4o Mini at 0.96 and 0.79, Gemini Flash 1.5 at 0.91 and 1.00, and Deepseek v3.2-exp at 0.90 and 1.00.

• Real-time latency: The model runs in 32.48 ms per sample on GPU and 161.16 ms on CPU, compared to 892 ms for Phi-3-mini on an A100. The paper states LLM Judge latency is 556–734 ms in related work, with the table listing Deepseek-v3.2-exp at 557 ms, Gemini-Flash-1.5 at 603 ms, and GPT-4o Mini at 735 ms. Isolation Forest on CPU is under 1.0 ms. The reported speedup is 17× over the fastest LLM judge (Deepseek) and 27× over Phi-3-mini, described as a conservative estimate because the model was benchmarked on a T4 GPU while Phi-3-mini used an A100.

• Long trajectories degrade performance: Precision drops on trajectories exceeding 10 steps. Reported F1 is 0.96 for 2–5 step trajectories versus 0.87 for 11+ steps, attributed to the fixed 128-dimensional GRU vector acting as an information bottleneck.

Methodology in Plain English

The authors framed the problem through three successive hypotheses.

First, they treated an anomalous trajectory as a point outlier: they mean-pooled step embeddings from a SentenceTransformer into one fixed-size vector per trajectory and ran unsupervised detectors (VAE, Isolation Forest, One-Class SVM). This failed (F1 < 0.70) because averaging washes out the few anomalous steps.

Second, they treated an anomaly as a context mismatch: they fine-tuned all-MiniLM-L6-v2 contrastively on (task, trajectory) pairs using MultipleNegativesRankingLoss so that valid pairs have higher cosine similarity. This reached F1 ≈ 0.82 but performed inconsistently across trajectory formats and ignored step order.

Third, the final architecture, Trajectory Guard, is a Siamese recurrent autoencoder with two towers. A task tower is an MLP projecting task embeddings to a 128-dimensional latent vector; a trajectory tower is a GRU encoder that compresses the step sequence into a "thought vector," with a GRU decoder reconstructing the original sequence. Training uses a hybrid loss: a triplet margin loss with in-batch negative sampling aligns task and trajectory vectors ("is this the right plan?"), while an MSE reconstruction loss teaches valid step ordering and composition ("is this plan coherent?"). The combined objective is the contrastive loss plus α times the reconstruction loss, with α = 0.5.

Training details: 5,752 trajectories, all-MiniLM-L6-v2 as base embedder, 20 epochs, batch size 16, Adam optimizer, learning rate 2×10⁻⁵, MLP task head projecting 384→128 dimensions, GRU hidden dimension 128, triplet margin 1.0. The anomaly threshold was chosen on a held-out validation set to maximize F1. Benchmarks used an NVIDIA T4 GPU and an Intel Xeon @ 2.00GHz CPU.

Data construction: the training pool comprised 6,767 "good" trajectories from Galileo (the adaptive_tool_use configuration of galileo-ai/agent-leaderboard-v2, natural-language commands across enterprise domains such as banking and telecom) and AgentAlign (the "benign" category, structured JSON tool calls). That pool was split 85/15 into 5,752 training and 1,015 validation trajectories. Anomalies for training, validation, and the synthesized test portion were generated by prompting openai/gpt-5 to inject a random number (1, 2, or 3) of invalid steps into good trajectories, covering contextual anomalies (steps logical in isolation but out of context, such as "Search for new applications" in a telecom task or "CloseMusicApp" mid-workflow) and structural anomalies (malformed steps such as tool calls with risky arguments). The final test set contains 5,822 trajectories held out before any training: 1,000 good samples (250 Galileo, 750 AgentAlign) and 4,822 anomalous samples (836 synthesized — 248 Galileo, 588 AgentAlign — plus 3,802 RAS-Eval security benchmark trajectories and 184 Who&When logs drawn from 127 LLM multi-agent systems).

Why This Matters

The paper positions lightweight trajectory validation as a practical safety layer for agentic AI. It quantifies a latency-versus-accuracy tradeoff that matters for production: LLM judges reach high F1 (up to 0.95) but at 556–734 ms, which the authors deem unsuitable for real-time use, whereas a 32.48 ms guard can run inline.

Impact on research: It provides evidence that generic embedding-space anomaly detection does not transfer to agent plan validation, and it frames trajectory anomalies as two distinct failure modes — contextual and structural — that a single model can address jointly. It also contributes labeled evaluation setups combining synthetic perturbations with real failure logs.

Real-world applications:

• Inline safety screening of agent action plans before execution in production agent deployments.
• Security auditing of tool-calling agents, using RAS-Eval-derived vulnerabilities as a stress test.
• Monitoring multi-agent systems for failures attributable to specific error steps, using Who&When-style logs.
• Enterprise workflow automation (e.g., banking and telecom domains represented in Galileo), where out-of-context steps such as an unrelated application-search command could disrupt an account or service task.

Industry relevance: The 17–27× speedup and sub-100 ms inference suggest the model can run as a cheap guard alongside more expensive LLM judges, or screen first and escalate uncertain cases.

Future Directions

• Incorporate attention mechanisms: The authors identify the fixed 128-dimensional GRU encoding as a bottleneck for long sequences and propose attention for better long-range handling, targeting the reported drop from F1 0.96 at 2–5 steps to 0.87 at 11+ steps.
• Validate on more real anomalies: The paper notes reliance on GPT-5-synthesized anomalies risks circularity and calls for broader validation on real-world anomalies beyond the human-annotated hold-outs.
• Broaden domain training: The authors propose training across a wider range of domains to improve generalizability.
• Benchmark against quantized LLMs: The paper suggests comparing against quantized LLM judges to position the approach within evolving AI safety standards.

Target Audience

Researchers and practitioners working on LLM agent safety, agent observability, and runtime guardrails will benefit most, along with engineers who need to deploy low-latency validation in production agent systems. Readers interested in sequence anomaly detection and Siamese recurrent architectures will also find the hybrid-loss ablation relevant. Familiarity with embeddings, autoencoders, and contrastive learning makes the paper substantially easier to follow.

Authors’ abstract

Autonomous LLM agents generate multi-step action plans that can fail due to contextual misalignment or structural incoherence. Existing anomaly detection methods are ill-suited for this challenge: mean-pooling embeddings dilutes anomalous steps, while contrastive-only approaches ignore sequential structure. Standard unsupervised methods on pre-trained embeddings achieve F1-scores no higher than 0.69. We introduce Trajectory Guard, a Siamese Recurrent Autoencoder with a hybrid loss function that jointly learns task-trajectory alignment via contrastive learning and sequential validity via reconstruction. This dual objective enables unified detection of both "wrong plan for this task" and "malformed plan structure." On benchmarks spanning synthetic perturbations and real-world failures from security audits (RAS-Eval) and multi-agent systems (Who\&amp;When), we achieve F1-scores of 0.88-0.94 on balanced sets and recall of 0.86-0.92 on imbalanced external benchmarks. At 32 ms inference latency, our approach runs 17-27$\times$ faster than LLM Judge baselines, enabling real-time safety verification in production deployments.