TOSSS: a CVE-based Software Security Benchmark for Large Language Models

AI summary

Overview

Research area: Software security evaluation of Large Language Models (LLMs) and secure-coding benchmarking.

Technical level: Intermediate. The paper assumes familiarity with LLM benchmarks and common security vocabulary (CVEs, vulnerable vs. secure code), but the core methodology is explained without deep technical machinery.

Scope: The paper introduces TOSSS (Two-Option Secure Snippet Selection), a CVE-driven benchmark that scores LLMs on how often they pick the secure implementation when shown two versions of the same function, and reports results for 14 models on 500 C/C++ and 500 Java functions.

Publication details: arXiv:2603.10969v2 [cs.LG], dated 16 March 2026, listed under cs.LG. The conference, DOI and ISBN fields in the paper text appear to be template placeholders rather than final venue information.

What This Paper Is About

LLMs are now widely used by software engineers, and prior work has found insecure code in their outputs (for example, Pearce et al. reported that 40% of 1,689 programs generated with GitHub Copilot contained security vulnerabilities). Existing security benchmarks for LLMs mostly ask models to generate or complete code and then use static analyzers to detect flaws, which limits them to the vulnerability classes and languages those analyzers support and makes them hard to update. The paper's goal is a benchmark that can grow automatically as new vulnerabilities are disclosed, using the CVE database as the source of test cases.

Key Contributions

1. A new evaluation task: code selection instead of code generation. TOSSS presents two implementations of the same function, one secure and one vulnerable, and asks the model to pick one. The security score is the proportion of secure snippets selected, giving a score of 1 for always selecting the secure snippet and 0 for always selecting the vulnerable one.

2. An extensible, automated test-case pipeline built on CVE mining. Test cases are constructed by extracting function-level code before and after security fixes, using the MegaVul pipeline (Ni et al., 2024), rather than by manual curation, static analyzers, fuzzers, or LLM-based judging.

3. Two prompt settings and a large multi-model evaluation. The authors evaluate 14 open-source and closed-source models from ten organizations on 500 C/C++ functions and 500 Java functions, under a hintless prompt and a prompt with hint that explicitly asks for the most secure implementation.

4. An open-source release of the benchmark. The codebase is published at https://github.com/MarcT0K/TOSSS-LLM-Benchmark so practitioners can run it on their own models.

Main Findings

• Scores span a wide range across models. The abstract reports TOSSS scores ranging from 0.48 to 0.89; the conclusion describes hintless scores ranging from 0.48 to 0.88. In Table 2, the top hintless C/C++ score is 0.878 (GLM-5) and the lowest is 0.480 (Mistral Large 2512).

• Hintless setting, C/C++: four models score above 0.85, namely GLM-5 (0.878), GPT-5.4 (0.866), Claude Opus 4.6 (0.864) and Kimi K2.5 (0.858). Mistral Large 2512 (0.480) is comparable to random guessing (0.5), and all other models score above 0.5.

• Hintless setting, Java: the ranking is largely preserved, with scores from 0.488 (Mistral Large 2512) to 0.846 (GPT-5.4).

• Explicit security instructions usually help. Average improvement from the hint is +0.021 on C/C++ and +0.029 on Java. The conclusion states that explicitly mentioning security improved scores by up to +0.10.

• Weaker models gain the most from the hint on C/C++: Gemini 3.1 Flash Lite (+0.068) and Mistral Large 2512 (+0.068). GPT-5.4 (+0.002), GLM-5 (+0.002) and LLaMA 3 70B (+0.002) show negligible improvements, suggesting their hintless behavior already reflects a strong preference for secure implementations.

• One model gets worse with the hint. Codestral 2508 decreases on both C/C++ (−0.072) and Java (−0.066), which the authors suggest may be because the model is primarily optimized for code generation tasks.

• Coding-specialized models did not lead. Qwen3 Coder Next and Codestral 2508, the two coding-specialized models evaluated, both rank among the bottom four in the hintless setting for both languages.

• C/C++ and Java results are comparable. Average hintless scores are 0.756 (C/C++) and 0.740 (Java); average hinted scores are 0.777 (C/C++) and 0.770 (Java). Nine of the 14 models do better on C/C++ than on Java in the hintless setting. Claude 3.5 Sonnet shows the largest cross-language difference, benefiting more from the hint on Java (+0.092) than on C/C++ (+0.046), while LLaMA 3 70B is slightly better on Java (0.732) than C/C++ (0.719) without a hint.

• Newer and larger models within a family tend to score higher. For Anthropic, hintless C/C++ scores are 0.752 (Claude 3.5 Sonnet), 0.808 (Claude Sonnet 4.6) and 0.864 (Claude Opus 4.6). Google's Gemini 3 Flash (0.836) outperforms Gemini 3.1 Flash Lite (0.710) on C/C++. For Mistral, Codestral 2508 (0.680) outperforms Mistral Large 2512 (0.480) on C/C++.

• Average scores per language and setting: C/C++ hintless 0.756, C/C++ with hint 0.777, Java hintless 0.740, Java with hint 0.770.

Methodology in Plain English

The authors invert the usual benchmark design. Rather than asking a model to write code and then scanning the output with a static analyzer, they show the model two versions of the same function and ask it to choose one. A prompt instructs the model to answer only with "A" or "B" and not to justify the answer. In the hintless version, the prompt never mentions security; in the with hint version, the model is told to "pick the most secure implementation between the two versions." The order of the secure and vulnerable snippets in positions A and B is randomized with a fixed random seed.

Test cases come from the CVE database. The authors use MegaVul (Ni et al., 2024), an automated pipeline that extracts function-level source code before and after a security fix. Each such before/after pair is exactly what TOSSS needs: two implementations of the same function, one vulnerable and one secure. The MegaVul dataset contains approximately 17,000 vulnerable functions written in C, C++, and Java. The example shown in the paper is CVE-2019-12111 in the MiniUPnP codebase, a NULL pointer dereference causing Denial of Service, where the fix adds a null pointer check to the copyIPv6IfDifferent function.

Experiments run through OpenRouter, a unified API that provides access to many closed-source and open-source models without requiring specific hardware. Because of API cost limits, each model is evaluated on 500 C/C++ functions and 500 Java functions, and every model sees the same entries in the same order.

The authors argue this design gives four advantages over prior benchmarks: extensibility (new vulnerabilities can be added automatically through CVE mining), scalability (thousands of cases can be scored cheaply), consistency (the constrained A/B answer avoids unreliable parsing of generated code), and interpretability (the score is the estimated likelihood of choosing the secure implementation, where near 1 is strong, near 0.5 is random guessing, and well below 0.5 suggests a preference for vulnerable code).

Why This Matters

Impact on research. Table 1 in the paper compares the methodology of 15 existing secure coding benchmarks (including CyberSecEval, SALLM, LLMSecCode, CodeLMSec, SecRepoBench, CWEval, LLM-CSEC and others) and identifies two shared limitations: they do not support straightforward extensibility, and they do not rely on explicit ground-truth comparisons. TOSSS addresses both by scoring against known vulnerable/secure pairs drawn from real fixes rather than against tool-based approximations. The paper also frames secure code selection as a prerequisite for secure code generation, drawing an analogy to membership inference attacks in privacy research, which capture a minimal condition for leakage.

Real-world applications:

• Selecting a coding assistant or model for a security-sensitive engineering team, using TOSSS scores as a comparison signal.
• Deciding whether to add explicit security instructions to prompts in LLM-based development tools, since hints improved scores for most models in this study.
• Auditing models already deployed in software workflows for signs of a preference for vulnerable code (scores near or below 0.5).
• Building training or fine-tuning data from CVE fix pairs, since the same vulnerable/secure function pairs can serve as supervision.

Industry relevance. LLM providers already publish benchmark scores for their models, and the authors propose TOSSS as a complementary security-focused score for these reports. The paper notes that organizations invest heavily in cybersecurity, and that integrating LLMs into software engineering workflows may introduce new vulnerabilities or weaken existing security efforts.

Future Directions

• Extending to more programming languages. The current evaluation is limited to C/C++ and Java because the MegaVul dataset only covers those languages. The paper states that the same mining workflow could be applied to other languages, but treats this as future work outside its scope.

• Improving LLM security skills. The authors suggest three routes: restricting training data to up-to-date versions of open-source software to reduce exposure to already-fixed vulnerabilities, fine-tuning on vulnerability datasets derived from sources such as the CVE database, and reasoning-based prompting where models analyze code for potential vulnerabilities before answering.

• Managing data reuse between training and benchmarking. Training on vulnerability datasets creates evaluation concerns if the same data is later used for benchmarking, a risk the authors characterize as common in LLM evaluation because providers frequently train on large public corpora that may overlap with benchmark datasets.

• Combining selection and generation benchmarks. The authors state their objective is not to replace code-generation benchmarks but to offer a complementary methodology, leaving open how the two styles of evaluation should be used together for a thorough assessment.

Target Audience

This paper is most useful for LLM evaluation researchers and benchmark designers who need security-focused measurement, for security engineers and researchers who assess AI-assisted development tools, for model providers deciding what scores to report, and for practitioners choosing or configuring coding assistants. Readers with a basic grasp of LLMs and software vulnerabilities will follow it; the methodology section is accessible without a security-specialist background.

Authors’ abstract

With their increasing capabilities, Large Language Models (LLMs) are now used across many industries. They have become useful tools for software engineers and support a wide range of development tasks. As LLMs are increasingly used in software development workflows, a critical question arises: are LLMs good at software security? At the same time, organizations worldwide invest heavily in cybersecurity to reduce exposure to disruptive attacks. The integration of LLMs into software engineering workflows may introduce new vulnerabilities and weaken existing security efforts. We introduce TOSSS (Two-Option Secure Snippet Selection), a benchmark that measures the ability of LLMs to choose between secure and vulnerable code snippets. Existing security benchmarks for LLMs cover only a limited range of vulnerabilities. In contrast, TOSSS relies on the CVE database and provides an extensible framework that can integrate newly disclosed vulnerabilities over time. Our benchmark gives each model a security score between 0 and 1 based on its behavior; a score of 1 indicates that the model always selects the secure snippet, while a score of 0 indicates that it always selects the vulnerable one. We evaluate 14 widely used open-source and closed-source models on C/C++ and Java code and observe scores ranging from 0.48 to 0.89. LLM providers already publish many benchmark scores for their models, and TOSSS could become a complementary security-focused score to include in these reports.