Temporal Attack Pattern Detection in Multi-Agent AI Workflows: An Open Framework for Training Trace-Based Security Models

AI summary

Overview

Research area: AI security for multi-agent LLM systems, specifically fine-tuning language models to detect attack patterns that only emerge across sequences of workflow steps (OpenTelemetry traces).

Technical level: Advanced — assumes familiarity with LLM fine-tuning (QLoRA, LoRA adapters, 4-bit quantization), evaluation harnesses, and observability formats.

Scope: A single-author paper that documents an end-to-end, openly released pipeline for building a trace-based agentic security classifier — from dataset curation and synthetic trace generation through three QLoRA training iterations, MCQA benchmarks, and a 30-trace practical validation — reporting a 31.4-point knowledge-benchmark gain alongside a 66.7% false positive rate.

What This Paper Is About

Existing LLM safety mechanisms judge individual text generations, so they miss malicious intent that only becomes visible across a multi-step agent workflow — a "list directory contents" call is benign alone but reconnaissance in an attack chain. Commercial AI security vendors run trace-based monitoring, but their training methods are closed, leaving practitioners without a reproducible way to build custom security models for their own threat landscapes. This paper provides the first openly documented methodology for fine-tuning a language model to classify OpenTelemetry workflow traces as benign, suspicious, or malicious, and reports both the gains and the severe limitations of that approach.

Key Contributions

1. First open methodology for fine-tuning LLMs on agentic workflow security: an end-to-end framework covering synthetic OpenTelemetry trace generation, dataset curation (80,851 examples from 18 sources), ARM64 training configurations, and deployment patterns, released publicly on HuggingFace.
2. Curated multi-source dataset plus synthetic trace generation: 45,825 examples from 18 public cybersecurity datasets (including AgentHarm, Agent-SafetyBench, PKU-SafeRLHF, BeaverTails, HaluEval, and TruthfulQA) combined with 35,026 synthetic OpenTelemetry traces covering multi-agent coordination attacks, stealth evasion, and regulatory violations (GDPR, HIPAA, PCI-DSS).
3. Lean iterative refinement on resource-constrained hardware: three QLoRA training iterations (V2 baseline, V3 with 111 OWASP-focused examples, V4 with 30 adversarial examples) on an NVIDIA DGX Spark ARM64 machine, improving accuracy from 42.86% to 74.29% (+31.43 points, McNemar's χ² = 18.05, p < 0.001, Cohen's h = 0.65), a 73.3% relative gain.
4. Empirical evidence that training data composition determines behavior: a roughly 90% attack-focused dataset produced a 66.7% false positive rate that an enhanced prompt (benign workflow guidance, indicator lists, two-shot examples) failed to change at all.

Main Findings

• Knowledge benchmark improvement is large and statistically significant: overall accuracy on the authors' 70-question custom cybersecurity MCQA rose from 42.86% (30/70) for the base Foundation-Sec-8B model to 74.29% (52/70) for V4. Agentic security accuracy went from 40.0% (8/20) to 70.0% (14/20); traditional security from 44.0% (22/50) to 76.0% (38/50). McNemar's test gave χ² = 18.05, df = 1, p < 0.001, with a 95% CI for the accuracy difference of [19.8%, 43.1%] and effect sizes of Cohen's h = 0.65 overall, 0.61 agentic, 0.66 traditional.

• Iterative targeted augmentation outperformed scale: on the custom MCQA, V2 scored 61.4% overall (50% agentic, 66% traditional), V3 reached 67.1% (+5.7 points; 65% agentic, +15 points; 68% traditional) using 111 OWASP and Microsoft Taxonomy examples, and V4 reached 74.3% (70% agentic, 76% traditional) using 30 additional adversarial examples. Overall progression was 61.4% → 67.1% → 74.3% (+12.9 points), with agentic accuracy up 20 points and traditional up 10 points.

• Standard benchmark scores: MMLU Computer Security reached 74.0% (±4.4% SE, 95% CI [65.4%, 82.6%], 100 questions, 5-shot, bfloat16, lm-eval-harness v0.4.9.2), and MMLU Security Studies reached 72.24%.

• Practical trace analysis fails badly: on 30 manually labeled traces (15 malicious, 15 benign), V4 (Q4_K_M quantized, 4.92 GB, temperature 0.1, top-p 0.95) achieved 30.0% overall accuracy (9/30), 60.0% true positive rate (9/15), 0.0% true negative rate (0/15), 66.7% false positive rate (10/15), 47.4% precision, 60.0% recall, and F1 = 0.529. The confusion matrix was TN = 0, FP = 10, FN = 6, TP = 9. Five benign and six malicious traces were labeled "SUSPICIOUS" and counted as incorrect.

• Prompt engineering cannot fix the bias: re-evaluating the same 30 traces with a substantially enhanced prompt (explicit guidance that most enterprise workflows are benign, detailed benign/malicious indicator lists, and two-shot examples) produced identical results — 30% accuracy, 60% TPR, 0% TNR, 66.7% FPR. The authors treat this negative result as more significant than a partial improvement would be.

• Detected versus missed attack classes: the model caught multi-agent coordination (agent-A(query_db) → agent-B(compress) → agent-C(upload_external)), credential harvesting with reconnaissance, API key extraction and session hijacking, and GDPR violations with backdoor installations. It missed data exfiltration (read_file(/etc/passwd) → http_request(attacker.com)) and gradual privilege escalation (read_only → write → execute → admin), classifying both as "SUSPICIOUS."

• Benign workflows misclassified as attacks: report generation (query_database → generate_pdf → email_report) flagged as "data exfiltration," system monitoring (check_disk_usage → log_metrics) flagged as "resource exhaustion attempt," data analysis (read_csv → calculate_statistics → generate_chart) flagged as "tool chaining with harmful intent," and a CI/CD pipeline (git_clone → run_tests → build → deploy_staging) flagged as "privilege escalation."

• The base model had uneven prior knowledge that fine-tuning rebalanced: 100% on the security_fundamentals subcategory but 0% on access control, 0% on incident response, and 20% on threat intelligence. After fine-tuning, access_control rose from 0% to 33.3%, security_operations from 28.6% to 71.4%, and threat_intelligence from 20% to 30%.

• Training stayed stable: V2 ran 1,500 steps over 6h 43m on 80,851 examples (0.148 epochs), reducing loss 85.99% (3.68 → 0.52); V3 and V4 ran 500 steps each (30 minutes each) and reached final loss 0.038. The authors report no catastrophic forgetting, supported by appendix error counts falling across V2 → V3 → V4 (for example, autonomous harmful actions 47 → 12 → 3, a 93.6% reduction, and GDPR violations 12 → 3 → 0, a 100% reduction).

• The evaluation is small and the paper says so: confidence intervals on the trace results are wide (±18% for accuracy at 95% CI), and the authors state that future work should expand to 100+ traces stratified by attack complexity.

Methodology in Plain English

The researchers start from a security-focused instruction-tuned model, Foundation-Sec-1.1-8B-Instruct (built on Llama 3.1, 8.03B parameters), chosen because it had never been trained on agentic AI security concepts — making it a clean baseline.

They build a training corpus in two parts. First, they collect and deduplicate 45,825 examples from 18 public cybersecurity and safety datasets, dropping 12.3% as redundant via instruction-text hashing and semantic deduplication. Second, because public datasets contain almost no labeled malicious agent workflows, they use Claude Sonnet 4.5 with templates to generate 35,026 synthetic OpenTelemetry traces — timestamped records of which agent called which tool, with what parameters and status codes — covering distributed attack chains across 2–5 agents, gradual privilege escalation, and GDPR/HIPAA/PCI-DSS violations, with attack patterns requiring 5–50 step context windows. Combined, this yields 80,851 examples.

They then fine-tune with QLoRA on an NVIDIA DGX Spark (Grace Blackwell ARM64, 128 GB memory): 4-bit NF4 quantization, rank-16 LoRA adapters, an 8-bit AdamW optimizer, batch size 8, and BF16 precision. Training proceeds in three rounds: V2 trains on the full 80,851 examples for 1,500 steps at learning rate 2e-4; V3 continues from V2's weights for 500 steps at a reduced 1e-4 with 111 examples drawn from OWASP Top 10 for Agentic Applications 2026 and Microsoft's Taxonomy of Failure Modes, targeting gaps found in an error analysis; V4 continues from V3's weights for another 500 steps with 30 adversarial examples.

Evaluation happens at three levels: MMLU Computer Security via lm-eval-harness, a custom 70-question MCQA benchmark spanning threat intelligence, vulnerability management, network security, application security, cryptography, and agentic AI security, and finally a practical test — classifying 30 real workflow traces with manually labeled ground truth. The third test is the one that exposes the problem, and the authors then run an ablation with a much more careful prompt to check whether inference-time instructions could compensate.

Why This Matters

Impact on research. The paper's most transferable result is negative and methodological: a model can score 74.29% on security knowledge questions covering the very attack concepts it is supposed to detect, and still misclassify two-thirds of benign production workflows. It also provides a concrete demonstration that inference-time prompt engineering cannot override representations learned from imbalanced training data — a finding the authors argue is more valuable than a partial improvement would have been. Combined with the full open release on HuggingFace (datasets, training scripts, configurations, benchmarks, and the 70-question MCQA and 30 labeled traces), it gives other groups a reproducible baseline where none previously existed.

Real-world applications:

• Security operations monitoring: the current model is usable for alerting and analyst review queues, but the authors explicitly state it is not suitable for automated blocking because of the 66.7% false positive rate.
• Compliance auditing: batch analysis (5–30 second latency per trace, offline processing of historical logs) maps onto forensic review of historical agent activity for GDPR, HIPAA, and PCI-DSS violations.
• Production guardrails for high-risk operations: a hybrid pattern at 100–200 ms latency that uses rule-based filters (file access patterns, external network calls, privilege changes) to pre-screen only suspicious operations before the model sees them.
• Governance alignment: the threat categories follow OWASP Top 10 for Agentic Applications 2026 and Microsoft's Taxonomy of Failure Modes in Agentic AI Systems, which both recommend behavioral detection for multi-agent coordination attacks, so the framework is aimed at organizations trying to implement those recommendations.

Industry relevance. The paper addresses a named proprietary gap: commercial vendors ship trace-based monitoring but do not publish their training methodology. This work targets practitioners who want custom models fitted to their own threat landscape instead of relying on general-purpose models. It also operates inside real constraints — training on a single ARM64 desktop-class machine rather than a cluster — which matters for teams without large GPU budgets, though the authors flag that the ARM64-specific workarounds (Triton compilation flags, bitsandbytes rebuilds) and the proprietary DGX Spark platform create reproduction barriers, with x86_64 CPU-only training 5–10× slower.

Future Directions

• Balanced retraining (V5): the authors propose 80K benign plus 80K malicious traces (160K total), 500–1K continuation steps at learning rate 5e-5, targeting at least 75% accuracy, at least 95% TPR, at least 65% TNR, and under 35% FPR, validated on a 2K holdout set with expert review.
• RAG augmentation as an alternative to retraining: a 10K+ benign workflow knowledge base supplied as runtime context, potentially reaching similar performance without a full retraining cycle.
• Larger and more rigorous evaluation: the paper calls for 500+ traces and 2K holdout sets, multi-seed training runs (n ≥ 3), cross-validation, human expert validation (n = 5, 500 traces, with a κ measurement), and head-to-head comparison against commercial models such as GPT-4 and Claude 3.5 Sonnet on identical benchmarks.
• Generalization and deployment testing: testing across novel attack patterns, different agent frameworks (LangChain, AutoGPT, custom), non-OpenTelemetry observability formats, alternative base models (Llama 3.3, Qwen 2.5), a 90-day production pilot across 2–3 enterprises, plus model compression and explainability work.

Target Audience

This paper is best suited for AI security engineers and applied ML practitioners who are building or evaluating runtime defenses for agentic systems and need a concrete starting point rather than a vendor white paper. It is also valuable for ML researchers studying data-centric fine-tuning, since the ablation on prompt engineering and the iterative V2/V3/V4 design illustrate how dataset composition shapes behavior. Security architects and risk/compliance leads will find the deployment patterns and the explicit human-in-the-loop recommendation directly actionable, while the limitations section is written for reviewers and replication teams who need to know exactly what has not yet been validated.

Authors’ abstract

We present an openly documented methodology for fine-tuning language models to detect temporal attack patterns in multi-agent AI workflows using OpenTelemetry trace analysis. We curate a dataset of 80,851 examples from 18 public cybersecurity sources and 35,026 synthetic OpenTelemetry traces. We apply iterative QLoRA fine-tuning on resource-constrained ARM64 hardware (NVIDIA DGX Spark) through three training iterations with strategic augmentation. Our custom benchmark accuracy improves from 42.86% to 74.29%, a statistically significant 31.4-point gain. Targeted examples addressing specific knowledge gaps outperform indiscriminate scaling. Key contributions include: (1) synthetic trace generation methodology for multi-agent coordination attacks and regulatory violations, (2) empirical evidence that training data composition fundamentally determines behavior, and (3) complete open release of datasets, training scripts, and evaluation benchmarks on HuggingFace. While practical deployment requires human oversight due to false positive rates, this work establishes the first reproducible framework enabling practitioners to build custom agentic security models adapted to their threat landscapes.