ShieldedCode: Learning Robust Representations for Virtual Machine Protected Code

AI summary

Overview

Research area: Software protection and reverse-engineering defense, combined with large language models for code and binary code representation learning (arXiv category: Natural Language Processing / cs.CL).

Technical level: Advanced. The paper assumes familiarity with transformer attention masking, contrastive learning objectives, Virtual Machine Protection (VMP), compiler optimization levels, and binary code similarity detection.

Scope: The paper proposes ShieldedCode, a protection-aware representation-learning framework that trains language models to generate, compare, and rank Virtual Machine (VM) protected code across multiple protection levels.

What This Paper Is About

Reverse engineering makes it easy to steal or tamper with software, and traditional Virtual Machine Protection defends against it using rigid, rule-based transformations whose instruction patterns are regular enough to be attacked by automated analysis. This paper treats software protection as a representation-learning problem: it builds paired datasets of source code and normalized VM implementations, then trains a model that preserves functional semantics while encoding how strong a given protection level is. The goal is a model that can generate protected code, tell whether two protected binaries came from the same source, and quantify relative protection strength.

Key Contributions

1. A learning-based framing of software protection. The authors formulate protection as a representation learning problem and introduce a protection-aware framework that aligns source code with VM-protected implementations across heterogeneous protection levels.
2. Modeling and objectives tailored to polymorphic VM code. They design hierarchical dependency modeling (intra-instruction, preceding-instruction, and inter-instruction levels) and combine functionality-aware and protection-aware contrastive losses so embeddings preserve semantic equivalence while capturing relative protection strength.
3. A protection effectiveness optimization (PEO) task. A new task quantifies and ranks VM variants derived from the same source, using hard-negative mining to separate highly obfuscated variants.
4. A two-stage training and evaluation pipeline. Continual pre-training (joint language modeling, contrastive learning, and PEO in an alternating fashion) followed by fine-tuning on generation, with evaluation spanning VM code generation, binary code similarity detection, and a human reverse-engineering study.

Main Findings

• VM code generation beat GPT-4o at L0. ShieldedCode reaches 26.95% Pass@1 and 35.68% Pass@10 on L0 VM code generation on HumanEval_compile, versus 22.58% Pass@1 and 31.47% Pass@10 for GPT-4o. It also leads at higher protection levels: Pass@1 of 18.47 (L1), 19.23 (L2), 14.71 (L3), and Pass@10 of 27.94 / 29.71 / 22.83, against GPT-4o's 17.43 / 15.26 / 11.89 and 25.18 / 22.36 / 18.99.
• Binary similarity detection gains over strong baselines. On BinaryCorp-VirtualAssembly, ShieldedCode achieves Recall@1 of 0.488 (O0+L1), 0.306 (O1+L1), 0.309 (O2+L1), 0.272 (O0+L3), 0.344 (O1+L3), and 0.469 (O2+L3), with MRR from 0.397 to 0.575. For contrast, jTrans (Linear Probe) scores 0.333 Recall@1 on O0+L1 and jTrans (Zero Shot) 0.137. The paper states ShieldedCode surpasses jTrans (Zero Shot) by 20.4% averaged Recall@1 and 16.3% MRR, and outperforms Trex by 15.5% averaged Recall@1 and 8.7% MRR. The abstract describes a 10% Recall@1 improvement over state-of-the-art methods such as jTrans, while the introduction describes 15.5 percentage points over jTrans (Linear Probe) on O0+L1 (0.488 vs 0.333) — the paper reports both figures.
• Long-input models also benefit. Applied to Granite 3B Code 128K, ShieldedCode's approaches raise the four-level average Pass@1 to 17.91 and Pass@10 to 25.25, versus 12.84 and 19.41 for standard fine-tuning and 4.62 and 6.44 for the base Granite model.
• Both contrastive training and polymorphic generation contribute. The variant without contrastive training or polymorphic generation (ShieldedCode-CL-PG) averages 15.78% Pass@1 and 27.41% Pass@10. Adding polymorphic generation to the contrastive variant raises the reported overall Pass@1 from 21.86% to 25.17% and Pass@10 from 35.25% to 38.30% (the text's Pass@1 figures differ from the arithmetic mean of the per-level values listed in the same table).
• Protection effectiveness optimization leads across pool sizes. Using K = 50, 100, 200, 500 on six real-world projects (Binutils, Curl, ImageMagick, SQLite, OpenSSL, Putty), ShieldedCode achieves the highest Recall@1 in all four K settings, beating prior methods by 2–5% on average; it ties with CodeArt when K = 500, and at K = 50 it outperforms DiEmph on four benchmarks while DiEmph leads only on SQLite.
• Resistance to reverse engineering improved in a human study. With 12 graduate students in computer security plus 3 professional reverse engineers for validation, analyzing 20 VMP-protected functions under a fixed budget of 30–45 minutes per function, ShieldedCode-protected code took 14.7 ± 5.3 hours to reverse on average, with a 17% success rate and a 2.1/10 confidence score. For comparison, VMProtect 3.7 took 3.4 ± 1.2 hours (67% success, 5.8/10), Themida 3.1 took 3.9 ± 1.5 hours (58%, 5.3/10), and unprotected code took 12.3 ± 4.1 min (100%, 9.2/10). Reported times reflect successful reversals only.
• Automated attacks struggled more. Pattern matching achieved a 0% success rate against ShieldedCode due to its polymorphic generation, while traditional VMP solutions gave up 43–61% of patterns. Symbolic execution with angr showed path explosion of 1,843× versus 127× for VMProtect and only a 3% completion rate versus 31% for VMProtect.

Methodology in Plain English

The authors start by building training data: a C source unit is compiled at optimization levels O0–O3, passed through a commercial VMP tool, and disassembled back into VM code. Each pair (c, N(vm)) applies a normalization operator with four canonicalization steps: removing debug symbols and comments, inserting whitespace around virtual instruction delimiters to stabilize tokenization, substituting virtual addresses with symbolic references, and replacing each instruction address with a canonical label such as [VINST-1] appended at the end of the instruction.

To model the structure of virtualized code, they add a hierarchical attention mask. Every virtual instruction is summarized by a [VINST]t marker token (intra-instruction level), each token can also see the marker of the immediately preceding instruction (preceding-instruction level), and each token connects to all earlier instruction markers (inter-instruction level). This replaces the flat causal mask used in standard language models.

Training combines several objectives. A standard language modeling loss is paired with functionality contrastive learning (FCL), which pulls embeddings of the same function together across source code and protection levels L0–L3 using weights exp(-|s-t|/τ) so that nearby protection levels are aligned more strongly, and protection contrastive learning (PCL), a soft-margin term that requires embedding distance to grow linearly with protection-level separation. These are combined as L_vmp = L_lm + λ(L_fcl + L_pcl). A separate protection effectiveness optimization (PEO) objective trains the model to rank the correct same-source candidate highest among K candidates at a different protection level, with hard negatives weighted by difficulty rank. Pre-training starts from CodeLlama 34b, applies polymorphic generation to half of the attention heads, and alternates between the L_vmp and L_peo tasks; fine-tuning optimizes only L_vmp. Generation evaluation uses HumanEval_compile (1000+ C functions compiled at O0–O3 and virtualized to x86 assembly, not used in training), sampling 20 VMP generations per function at temperature 0.2 and top-p 0.95 and reporting Pass@1 and MRR.

Why This Matters

The paper reframes software protection as something a model can learn and reason about, rather than a fixed set of transformation rules. It introduces protection-level-aware embeddings and the PEO ranking task, which gives defenders a quantitative way to compare obfuscation schemes. It also connects LLM-for-code research to binary analysis, extending prior work on assembly, LLVM-IR, and binary-to-source recovery (CodeArt, LLMCompiler, LLM4Decompile, Nova) into the harder regime of polymorphic VMP bytecode, where virtual-register renaming and interpreter-driven semantics break structural stability.

Real-world applications:

• Choosing protection configurations. Developers could rank candidate VM variants for the same function by predicted protection strength before shipping.
• Binary similarity and clone detection. Stronger embeddings help identify whether two binaries derive from the same source, useful for license/IP violation detection and patch analysis.
• Malware and obfuscation analysis. The same representation could assist analysts in clustering or attributing protected samples.
• Benchmarking protection tools. Vendors and buyers could compare commercial protectors (such as VMProtect 3.7 and Themida 3.1, as used in the study) with a learned, task-based metric instead of anecdote.

Industry relevance is direct: commercial VMP solutions are described as costly and limited in scope, while the paper's human study suggests measurable gains in analyst time and success rate. The authors also note that the model is processor agnostic in design, illustrated here on C programs compiled for x86-64.

Future

Authors’ abstract

Large language models (LLMs) have achieved remarkable progress in code generation, yet their potential for software protection remains largely untapped. Reverse engineering continues to threaten software security, while traditional virtual machine protection (VMP) relies on rigid, rule-based transformations that are costly to design and vulnerable to automated analysis. In this work, we present the first protection-aware framework that learns robust representations of VMP-protected code. Our approach builds large-scale paired datasets of source code and normalized VM implementations, and introduces hierarchical dependency modeling at intra-, preceding-, and inter-instruction levels. We jointly optimize language modeling with functionality-aware and protection-aware contrastive objectives to capture both semantic equivalence and protection strength. To further assess resilience, we propose a protection effectiveness optimization task that quantifies and ranks different VM variants derived from the same source. Coupled with a two-stage continual pre-training and fine-tuning pipeline, our method enables models to generate, compare, and reason over protected code. Extensive experiments show that our framework significantly improves robustness across diverse protection levels, opening a new research direction for learning-based software defense. In this work, we present ShieldedCode, the first protection-aware framework that learns robust representations of VMP-protected code. Our method achieves 26.95% Pass@1 on L0 VM code generation compared to 22.58% for GPT-4o., and improves binary similarity detection Recall@1 by 10% over state of art methods like jTrans.