VX-One: Typed, Calibrated Decision Heads for Real-Time Perimeter Triage - An empirical study of unknown-attack honesty, cascade gray zones, and calibration under drift

AI summary

Overview

Research area: Network security machine learning — intrusion detection, malware classification, probability calibration, and out-of-distribution/abstention behavior at the network perimeter.

Technical level: Advanced. The paper assumes familiarity with intrusion-detection benchmarks (UNSW-NB15, CIC-IDS2017, EMBER2024), signature engines (Suricata, Snort, Zeek, YARA), and calibration metrics (ECE, MCE, Brier, NLL, FPR@95%TPR), and it is written as a workshop draft for a Security & ML venue.

One-sentence scope: An empirical study of a small multi-head network (VX-One, 100K–10M parameters) whose outputs are three typed decision primitives — a calibrated gate, a closed-set choice that always includes an explicit unknown option, and an ordinal severity score — evaluated against calibrated classical baselines and inside a two-phase cascade with Suricata on three public benchmarks.

What This Paper Is About

Perimeter triage faces a trade-off: signature engines are fast and format-safe but structurally blind to unseen attacks, while generative and LLM-style systems are semantically rich but slow and prone to unparseable output. In late 2026 a commercial class of "System One" decision APIs (TypeSafe's Jev, Microsoft-Decision-1) began selling typed decision primitives with no peer-reviewed academic treatment or independent security evaluation. The paper builds those primitives directly as a small model, then measures — honestly and with negative results included — where typed, calibrated decisions actually help network triage and where they fail, including under temporal drift and on attack families the model never saw.

Key Contributions

1. A typed decision engine. A shared pre-norm MLP trunk with three primitive heads — a calibrated sigmoid gate, a choice softmax over a declared closed set that always contains unknown, and an ordinal score head implemented as CORN conditional ordinal regression over four severity levels (benign, low, high, critical) — trained multi-task with a modified homoscedastic weighting scheme and per-head temperature calibration on validation data. Verified parameter counts on UNSW-NB15 are 97K / 914K / 5.9M for the Nano / Small / Base scales, enforced inside a hard budget by unit test.

2. Measured unknown-attack honesty. Leave-one-attack-family-out evaluation across 9 UNSW-NB15 and 7 CIC-IDS2017 families, plus 4 months of EMBER2024 family drift, reporting unseen-family detection (gate AUROC) and attribution honesty (how often unknown absorbs what the model should not claim to know) as separate quantities.

3. A measured gray zone for the two-phase cascade. Suricata 7 with 52,674 live Emerging-Threats rules over flow-reconstructed CIC-IDS2017 traffic resolves only ~0.1% of daily flows under payload opacity, so the gray zone is ~99.9% of traffic; Phase 2 (VX-One) recovers it with combined recall ≥ 0.978, at a measured precision cost on web-attack days where Phase 1 is entirely blind.

4. Negative results, reported. Calibration collapses under temporal drift for every method (ECE 0.04 → 0.40); a fixed abstention threshold is not portable across benchmarks; class-weight tilts that help weak-label attribution can hurt on saturated benchmarks; removing the unknown option raises macro-F1 by up to +11.2 points, quantifying the honesty/utility trade-off.

Main Findings

• Detection is won, attribution is lost. VX-One Small tops AUROC and FPR@95%TPR on UNSW-NB15 (0.9834 AUROC vs. XGBoost's 0.9818; 0.1211 FPR@95 vs. 0.1369) and under temporal drift on CIC-IDS2017 by-day (0.7112 vs. 0.6372). But calibrated XGBoost wins multi-class attribution on all three benchmarks — UNSW macro-F1 0.4968 vs. 0.4571, stratified CIC 0.9416 vs. 0.8826, EMBER2024 0.8094 vs. 0.7549. The authors state the inversion rather than hide it.

• Drift breaks trees' operating points entirely. On CIC by-day, XGBoost, Random Forest and the MLP all hit FPR@95 = 1.0000 — no threshold reaches 95% TPR at nontrivial FPR — while VX-One retains a partially usable operating point (0.5731).

• Saturation makes stratified CIC a sanity check, not a result. All methods land near AUROC 1.0 with ECE ≤ 0.001 on the stratified split; the same dataset becomes hard under by-day splitting. The paper argues headline "F1 0.94" numbers on IDS2017-stratified measure fitting to re-used capture artifacts.

• The gate generalizes to unseen families on UNSW, 9 of 9 folds. Leave-one-family-out gate AUROC ranges from 0.8519 (fuzzing) to 0.9992 (analysis), with abstention on the excluded family ramping from 3.0% to 92.0%, while macro-F1 on known families stays at 0.38–0.56.

• On CIC the failures are structural. The gate drops below chance on recon (0.4259) and near chance on infiltration (0.5329, n = 5) — in 77 flow-statistics features, a datacenter port scan looks like a benign port-touching client. The authors state explicitly that reconnaissance is a temporal-aggregation phenomenon invisible to per-event statistics.

• A fixed abstention threshold does not transfer. On UNSW, τ_abs = 0.5 abstains 3–92% on excluded families; on saturated CIC the same threshold abstains on zero rows, and 94.5% of unseen-family events are absorbed into normal at saturated confidence (median max-probability ≥ 0.999 on five of seven CIC folds, 1.000 on recon). Sweeping τ_abs from 0.05 to 0.99 gives UNSW an operating frontier (τ = 0.99 → 97.3% abstention with 0.66% false attribution) but gives CIC no operating point (17.8% abstention at best). Margin-based selective classification fails for the same reason (median margin 1.000 on recon).

• Trained-unknown absorbs drift on EMBER2024. With training on weeks 0–45, gate AUROC is 0.997 and FPR@95 0.012. berbew, which grew from 34.5K to 92K test rows, retains recall 0.998; the mutated wacatac falls to recall 0.129 with 85% routed to unknown; grandoreiro (n = 226) falls to 0.221 with 78% routed to unknown.

• Latency is inline-scale. Single-event, batch=1, single-thread CPU, 3,000 timed events: p50 675 µs (Nano), 955 µs (Small), 1,652 µs (Base); p99 773 / 1,022 / 2,082 µs.

• The cascade's gray zone is the headline number. Over five days of CIC-IDS2017, combined cascade recall is ≥ 0.978 and precision ≥ 0.95 on every day except Thursday (0.552). Thursday — 458,483 flows, 2,216 attacks — is the failure case: Phase 1 alone reaches only 0.059 precision without payload, and the Phase-2 false-positive rate across roughly 456K benign rows dominates.

• Risk–coverage shows the cascade gating's limits. Monday (benign-only) is a null point where every adjudication is an error; Friday admits 0.416 of flows at risk ≤ 5% with the plateau supplied entirely by Phase 2 (Phase 1 adjudicates 0.0004 of flows at precision 0.067); Thursday admits nothing at risk ≤ 10%, and Phase 2's clean core covers only 0.44%.

• Calibration is a probability fix, not a ranking fix. Removing temperature scaling leaves AUROC and argmax untouched while ECE on UNSW moves from 0.0496 to 0.0980; classwise calibration of the Choice head stays ≤ 0.007 on all four main runs.

• Kitsune: calibration creates no signal. Platt-calibrated on validation, Kitsune is well calibrated (ECE 0.037) but near-chance on discrimination (AUROC 0.6086 on UNSW).

• Ablations are regime-dependent. Removing the text branch costs roughly 5 macro-F1 points on UNSW (0.4571 → 0.4079) and is an identity on EMBER2024 by construction; removing class tilts improves saturated CIC attribution by +4.9 macro-F1; zero dropout costs −14.7 macro-F1 on CIC but gains +2.7 on EMBER; removing the trained unknown option raises EMBER macro-F1 by +11.2 points.

• The commercial System One API does not transfer zero-shot. On 2,021 stratified UNSW test rows (1,000 benign, 1,021 attack), the vendor gate is at chance (AUROC 0.568, FPR@TPR95 0.932, ECE15 0.265), choice macro-F1 is 0.069 vs. 0.534 for VX-One, and severity MAE is 0.809 vs. 0.333. Latency is 208 ms p50 / 591 ms p95 at roughly 2.2M input tokens (US$0.09). Adding an explicit unknown option leaves it unchanged or slightly worse (AUROC 0.476, macro-F1 0.066, unknown assigned to only 0.35% of rows).

• Composition bounds matter for controller design. The paper proves that single-head post-processing keeps calibration error bounded by that head's own deficiency, but two individually well-calibrated heads can compose into a joint policy with worst-case calibration error of 1/4 — so the controller composes boolean decisions between gates and never probabilities between heads.

• Seed variance confirms detection conclusions only. Re-runs with seeds {42, 43, 44} reproduce the seed-42 UNSW row to the fourth decimal; AUROC moves ≤ 0.002 except by-day CIC (0.734 ± 0.018). Macro-F1 is stable on UNSW (0.456 ± 0.001) and EMBER (0.746 ± 0.012) but not on CIC (stratified 0.831 ± 0.074; by-day 0.344 ± 0.085, spread 0.24–0.45).

Methodology in Plain English

Each telemetry event — one network flow or one binary's features — is compiled into a fixed-size numeric state (39 dimensions for UNSW-NB15, 77 for CIC-IDS2017, 2,568 for EMBER2024), plus lookup embeddings for protocol-level categorical fields where the benchmark supplies them. Features are median-imputed, signed-log-transformed, robustly scaled by training median/IQR, and clipped to ±8.

That state goes through a small residual MLP trunk into three heads whose outputs have declared types: a probability that the event is an attack, a probability distribution over a fixed list of named options that always includes unknown, and a cumulative distribution over four ordered severity levels. All three are trained together with one combined loss, and each head's probabilities are rescaled afterwards with a single temperature fitted on validation data.

The evaluation is designed to avoid strawmen: XGBoost (500 trees, hist mode), Random Forest (300 trees), a 2×256 MLP and a flow-adapted Kitsune all get the same feature pipeline, the same class weights where analogues exist, and the same temperature calibration. The authors also report results both raw and calibrated. For unseen-attack honesty, they hold out one attack family at a time (leave-one-family-out) so the model never sees it in training. For the cascade experiment, they reconstruct five days of CIC-IDS2017 into PCAPs without payload, run Suricata 7.0.10 with 52,674 Emerging-Threats rules plus 8 structural rules of their own, and measure what Phase 1 catches versus what Phase 2 must handle — explicitly framing this as a payload-blind lower bound, not a real-world gray-zone estimate. Combined cascade numbers rest on a documented independence assumption at the shared-flow level. Everything is reproducible from frozen JSON configs with SEED=42, run on L4 serverless GPUs with metrics computed offline from dumped raw logits, and validated by 60 unit tests (60/60 green).

Why This Matters

Impact on research. The paper formalizes and publicly evaluates a commercial product category ("System One" decision models) that previously had only vendor-reported benchmarks — including claims like 193.6× faster and 444.6× cheaper that the paper treats as industry claims, not evidence. It also demonstrates a methodological point that generalizes beyond this system: saturation is a property of a split, not a dataset, and stratified IDS2017 results should be reported alongside by-day results. The negative results (calibration collapsing to ECE 0.40 under drift, non-portable abstention thresholds, the +11.2-point cost of unknown) give the field concrete failure modes to design against.

Real-world applications:

• Perimeter triage with encrypted or uninspectable traffic. The payload-blind cascade experiment directly models environments with TLS everywhere and no payload inspection in managed switches, where signature engines resolve ~0.1% of flows.
• Mitigation controllers that need thresholdable confidences. Because every output is typed and calibrated, a policy engine can set a detection threshold, an abstention threshold, and a severity cutoff as explicit cost statements rather than tuned constants — the paper maps a 10:1 miss:alarm cost ratio to a threshold of approximately 0.091.
• SOC alert triage and advisory-mode blocking. The Thursday result argues for Phase 2 in advisory mode (block with abstention) rather than auto-drop when the blind phase's false positives are multiplied by large volumes

Authors’ abstract

Perimeter triage faces a latency/semantics trade-off: signature engines are fast and format-safe but blind to unseen attacks, while generative models are semantic but slow and produce unparseable output. In late 2026 a class of commercial System One decision models (TypeSafe Jev; Microsoft-Decision-1) began selling typed decision primitives - binary gates, closed-set choices, ordinal scores - as product APIs, with no academic treatment. We present VX-One: a small multi-head network (100K-10M parameters) whose outputs are these primitives - a calibrated gate, a choice over a closed candidate set that always includes an explicit unknown option, and an ordinal score, trained multi-task, temperature-calibrated per head (~0.7-1.7 ms p50 on CPU). We evaluate it on UNSW-NB15, CIC-IDS2017, EMBER2024 against calibrated XGBoost, Random-Forest, MLP, Kitsune, and measure a two-phase cascade (Suricata with 52,674 Emerging-Threats rules -> VX-One) over five days of CIC-IDS2017. Findings: (i) VX-One wins detection (AUROC/FPR@95%TPR) on UNSW-NB15 and under temporal drift on CIC-IDS2017 (0.711 vs 0.637), but XGBoost wins multi-class attribution on all three benchmarks; (ii) leave-one-family-out gate detects never-seen families on 9/9 UNSW folds yet drops below chance on CIC recon family, and abstention threshold transfers poorly across benchmarks; (iii) on payload-less reconstructions signature engines resolve ~0.1% of flows - gray zone ~99.9% of traffic, recovered by Phase 2 (combined recall >=0.978); (iv) calibration collapses under temporal drift (ECE 0.04->0.40) for every method while ranking partially survives. We provide a measured account of where typed calibrated decisions help perimeter triage, and where they fail.