Latent Sculpting for Zero-Shot Generalization: A Manifold Learning Approach to Out-of-Distribution Anomaly Detection

AI summary

Overview

Research area: Machine learning for network intrusion detection — specifically out-of-distribution (OOD) and zero-shot anomaly detection using manifold learning, tabular Transformers, and normalizing flows, evaluated on the CIC-IDS-2017 benchmark.

Technical level: Advanced. The paper assumes familiarity with Transformer encoders, normalizing flows (Masked Autoregressive Flows, MADE blocks), change-of-variables density estimation, and deep SVDD-style compactness penalties.

Scope: The paper proposes and evaluates "Latent Sculpting," a two-stage architecture that first structures a latent space with a custom geometric loss and then fits a density model to that structured space, in order to detect attack classes deliberately withheld from training.

What This Paper Is About

Machine learning intrusion detection systems perform well on attacks they have seen but frequently fail on new ones, because their decision boundaries are tightly tied to the training distribution. The authors call this failure mode "generalization collapse": supervised models draw boundaries around known classes and leave open "negative space" where novel anomalies can sit undetected, while purely unsupervised models struggle with the multi-modal nature of network traffic and produce high false-positive rates. The paper's goal is to fix this by explicitly separating two jobs — learning the geometry of normal traffic and learning the probability density of normal traffic — so that unseen attacks become both geometrically and probabilistically distinguishable.

Key Contributions

1. Binary Latent Sculpting Loss: A new optimization objective that forces a Transformer-based tabular encoder to compress benign traffic into a dense, low-entropy hypersphere (via a compactness penalty adapted from Deep SVDD) while pushing known anomalies outside a learnable minimum-distance margin.
2. Two-stage manifold density estimation: The coupling of the sculpted encoder with a Masked Autoregressive Flow (MAF), which projects the structured benign manifold into a tractable probabilistic space for exact, threshold-based likelihood scoring of novel threats.
3. Zero-shot performance on CIC-IDS-2017: Using a protocol that withholds Bot, DoS Slowloris, DoS Slowhttptest, and Infiltration from training, the framework averages F1 = 0.980 ± 0.000 on known attacks and zero-shot OOD F1 = 0.867 ± 0.021 with AUROC 0.913 ± 0.010 at an 85th-percentile threshold across three seeds (0, 42, 1024). Ablations indicate both the sculpting loss and density estimation are needed for optimal performance.
4. Detection of stealthy and low-volume intrusions: Average recall of 78.7% (peaking at 97.2%) on Infiltration and over 94% on low-volume DoS variants, zones where the authors say conventional approaches often fail.

Main Findings

• Stage 1 alone collapses on stealthy OOD: The Tabular Transformer achieves an average Anomaly F1 of 0.980 on internal validation, but when exposed to unseen OOD data it reaches roughly 0% detection on withheld Bot and Infiltration attacks across all three seeds, because those vectors fall inside the learned benign margin.
• Stage 2 rescues zero-shot detection: Adding the MAF at the 85th-percentile threshold raises the full two-stage pipeline to an average OOD Anomaly F1 of 0.867 and average AUROC of 0.913, with OOD AUPRC of 0.882.
• Known-attack performance stays high: Internal average AUROC is 0.978 and AUPRC 0.979; internal anomaly recall is 1.000 with precision 0.843, and internal benign precision is 1.000 with recall 0.813.
• OOD metric table (Table I, γ85): Unseen benign precision 0.893, recall 0.820, F1 0.850; unseen anomaly precision 0.833, recall 0.900, F1 0.867.
• Per-seed variability on stealthy attacks: Infiltration recall goes from 0.0278 (Seed 0), 0.0278 (Seed 42), and 0.0000 (Seed 1024) at Stage 1 to 0.9722, 0.4444, and 0.9444 respectively in the final pipeline. Bot goes from 0.0000 in all three seeds to 0.6536, 0.4095, and 0.4100.
• Low-volume DoS is handled strongly: DoS Slowhttptest final recall is 0.9998, 0.8945, and 0.9444 across seeds, while DoS Slowloris final recall is 0.9991, 0.9822, and 0.9990.
• Averaged per-attack zero-shot recall (Table III): Bot 49.10%, DoS Slowhttptest 94.62%, DoS Slowloris 99.34%, Infiltration 78.70%.
• Comparison to flow-based baselines (Xu and Liu): Flow-level supervised networks in that baseline saw OOD F1-scores drop below 0.33, and their leading unsupervised alternative (OCSVM) managed 0.7575. The baseline MLP and CNN recorded 0.0000 recall on withheld Bot attacks, mirroring the paper's isolated Stage 1. The baseline MLP peaked at 0.3675 on DoS Slowloris, versus 0.9934 for Latent Sculpting.
• Comparison to a packet-level baseline (Matejek et al.): That raw-packet safeguard achieved 0.95–0.99 AUROC but detected OOD Infiltration at only 7.80%, Bot at 9.50%, and DoS Slowhttptest at 52.48%. For Heartbleed, the packet-level model fell from 0.4418 in-distribution to 0.0002 OOD, while Latent Sculpting maintained 1.0000 recall. The paper notes its own approach uses pre-aggregated flow metrics rather than raw byte arrays.
• Also strong on cataloged attacks: Web XSS detection reaches an average of 0.9926 versus the flow-level MLP's 0.0308, and Web SQL Inject reaches 0.9444 versus the baseline maximum of 0.2500. Note that in the Xu and Liu baseline setup, Infiltration was treated as a known training attack rather than withheld.
• Computational design claim: The rigid Stage 1 distance margin filters high-velocity attacks such as DDoS and DoS Hulk at the encoder layer, reserving the MAF for ambiguous edge cases. The paper states this is intended for real-time, line-rate viability, but does not report measured latency or throughput numbers.

Methodology in Plain English

The framework has two sequential stages.

Stage 1 — structuring the latent space. Network flow records are treated as a sequence of tokens, one token per feature, rather than a single flat vector. Each of the 71 input features is embedded into a 64-dimensional vector, combined with learnable positional embeddings so the model retains which feature is which, then passed through multi-head self-attention layers. Average pooling collapses the sequence into a single 64-dimensional latent vector.

Training uses a custom loss. Within each mini-batch, the model computes the centroid of the benign samples and the Euclidean distance of every sample from that centroid. A learnable margin (initialized to 5.0) and a learnable temperature (initialized to 1.0) convert that distance into a logit, which is fed into binary cross-entropy: benign samples are pulled inside the margin, anomalies are pushed outside it. A separate compactness penalty, applied only to benign samples, pulls them tightly toward the centroid. The total objective is the classification loss plus α times the compactness penalty, with α set to 0.05 empirically. The effect is a dense benign cluster surrounded by an empty geometric buffer.

Stage 2 — probabilistic scoring. Because the benign cluster is now compact and continuous, a Masked Autoregressive Flow can be fit to it. The flow learns an invertible mapping from the latent vectors to a standard Gaussian, allowing exact log-likelihood computation via the change-of-variables formula. It uses MADE blocks so the autoregressive property is enforced without looping over dimensions one at a time.

Inference. A new sample's distance from the benign centroid is compared to the learned margin. If it exceeds the margin, it is classified as an attack immediately, with no further computation. If it falls inside the margin, it goes to the flow, and is flagged as an attack only if its log-likelihood falls below a threshold γ set at the 85th percentile of benign validation scores.

Data handling. Two features were engineered: Bytes per Packet and Packets per Second (with ε = 10⁻⁶ added to denominators). Zero-variance columns were dropped, and the remaining 71 continuous features were standardized, with variance calculation and scaler fitting performed strictly on the training partition. Benign training samples were capped to match the frequency of the most prevalent anomaly class, DoS Hulk, at N = 184,804, yielding a training set of 620,283 samples (184,804 benign and 435,479 anomalous). Evaluation used strict 1:1 balancing for both the internal validation set (N = 217,740) and the OOD test set. Stage 1 uses 158,912 trainable parameters, 3 encoder layers, 4 attention heads, a feedforward dimension of 256, and AdamW with 0.1 dropout and a maximum gradient norm of 1.0; Stage 2 uses 16 flow layers with hidden dimension 512 and Adam. Both phases use batch size 512, 10 epochs, and a learning rate of 5 × 10⁻⁴.

Why This Matters

Impact on research. The paper reframes zero-day anomaly detection as a problem of latent-space topology rather than classifier capacity. Its central empirical claim — that a geometric margin alone collapses on OOD data but becomes effective once a density model is layered on the structured manifold — is a concrete, testable argument against the "one big supervised model" approach to intrusion detection. It also provides a direct methodological comparison point against both flow-level (Xu and Liu) and packet-level (Matejek et al.) baselines on the same benchmark.

Real-world applications:

• Enterprise and ISP network monitoring: Screening high-bandwidth traffic for previously uncatalogued intrusions, where the two-tier design means cheap distance checks absorb the bulk of high-volume attack traffic and expensive density evaluation is reserved for ambiguous cases.
• Critical infrastructure protection: Environments where stealthy, low-volume attacks such as Slowloris and Slowhttptest are a persistent concern and where the reported recall exceeds 89% at essentially all seeds.
• Security operations center triage: Because the model outputs calibrated probabilistic scores, alerts can be ranked rather than presented as a flat list, which supports analyst prioritization.
• Adversarial and OOD detection outside cybersecurity: The authors explicitly argue the geometric-isolation idea could apply to adversarial image perturbations, anomalous biometric telemetry, or out-of-distribution prompt injections and hallucinations in large language models.

Industry relevance. Network intrusion datasets are severely imbalanced and labeling zero-day attacks in advance is impossible, so a framework that performs well while withholding entire attack classes maps onto a real operational constraint. The paper also emphasizes a lightweight footprint and a hierarchical inference path that avoids running the flow model on most traffic. One caveat for practitioners: no measured latency, throughput, or deployment-cost figures are reported, only architectural arguments for efficiency.

Future Directions

• Semi-supervised Stage 1: Moving the structural optimization to a semi-supervised paradigm, using a small fraction of labeled data alongside large pools of unannotated traffic, to cut labeling costs while keeping rigid manifold boundaries.
• Cross-domain transfer: Applying the Binary Latent Sculpting objective to continuous, non-tabular domains, including adversarial image perturbations, biometric signal telemetry, and LLM guardrails against prompt injection or hallucination.
• Reducing seed sensitivity: Seed 42 produced notably lower zero-shot recall on Infiltration (0.4444) than Seeds 0 (0.9722) and 1024 (0.9444), and the paper attributes this to stochasticity of manifold structuring. Stabilizing this variance is an open question.
• Threshold selection: Results are reported at a fixed 85th-percentile threshold, which the authors note trades benign recall for aggressive anomaly recall. Whether the threshold should adapt to deployment conditions is not resolved in this work.

Target Audience

Researchers and graduate students working on anomaly detection, OOD generalization, or representation learning will find the latent-sculpting loss and the two-stage decoupling argument directly relevant. Practitioners building or evaluating network intrusion detection systems — particularly those working with CIC-IDS-2017 — will benefit from the benchmark comparisons against MLP, CNN, OCSVM, LOF, and the packet-level normalizing flow baseline. Readers interested in normalizing flows applied to structured, non-image data, or in adapting deep SVDD-style compactness penalties to new domains, will also find the methodology transferable. A working familiarity with Transformers and density estimation is assumed; the paper is not an introductory read.

Note on disclosure: The acknowledgments state that two authors were partially supported by a United States Department of Homeland Security Research Team Follow-up Grant, and that large language models were used to assist in drafting and refining the manuscript and in supporting development of PyTorch training and evaluation code. Source code is linked at https://github.com/Rajeeb321123/Latent_sculpting_using_two_stage_method.

Authors’ abstract

Detecting previously unseen attacks remains a major challenge for machine learning-based intrusion detection systems. Deep models trained on network traffic often achieve high accuracy on known attacks but fail under distributional shift because their decision boundaries are tightly coupled to the training data distribution. We introduce Latent Sculpting, a two-stage anomaly detection framework that improves robustness by explicitly structuring the latent representation before density estimation. The first stage trains a Transformer-based tabular encoder using a novel Binary Latent Sculpting loss, which encourages benign traffic to form a compact latent cluster while enforcing separation from anomalous patterns. The second stage fits a Masked Autoregressive Flow to the resulting latent space to produce calibrated probabilistic anomaly scores. Under a strict zero-shot evaluation protocol on the CIC-IDS-2017 benchmark, Stage 1 attains an F1-score of 0.98 on known attacks, while Stage 2 -- evaluated at the balanced threshold (85th-percentile) -- achieves a zero-shot OOD F1-score of 0.867 and AUROC of 0.913. The model successfully detects difficult distribution shifts including stealthy infiltration attacks (78.7% recall, peaking at 97.2%) and low-volume DoS variants (>94% recall), scenarios where conventional approaches often fail. Our results suggest that explicitly separating latent geometry learning from density modeling provides a stable approach for detecting zero-day cyber threats.