KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards

AI summary

Overview

• Research area: Natural Language Processing / LLM evaluation for cybersecurity tool use (natural-language-to-command-line translation).
• Technical level: Intermediate. No deep mathematical background is required, but readers should be comfortable with LLM benchmarking terminology (exact match, F1, SFT, RLVR).
• Scope in one sentence: The paper introduces KaliBench, a benchmark and dataset of 8,504 natural-language query–command pairs covering 1,642 Kali Linux tools, used to measure and improve how well LLMs turn analyst intent into exact, executable CLI commands.

What This Paper Is About

Existing cybersecurity benchmarks for LLMs mostly test what models know (multiple-choice or structured questions) or how they perform in end-to-end agentic settings such as Capture-the-Flag challenges. Neither directly measures whether a model can produce a correct, executable invocation of a specific security tool, which matters because cybersecurity work depends on unforgiving command-line interfaces where a wrong flag, a misordered argument, or a bad alias breaks execution. The authors build KaliBench to isolate that skill — tool selection plus argument construction — and to show it can also be used as a reward source for training without ever running the generated commands.

Key Contributions

1. A fine-grained benchmark for schema-free NL-to-CLI evaluation. KaliBench is presented as the first benchmark for evaluating LLMs on cybersecurity tool invocation in schema-free CLI settings, with 8,504 query–command pairs across 23 capability dimensions and five security phases, split into 5,000 evaluation and 3,504 training samples after deduplication.
2. A verified and executable dataset. Ground-truth commands are produced through a multi-stage pipeline combining LLM validation, sandboxed terminal execution in a Kali-linux-everything Docker container, and human-in-the-loop review, ensuring both semantic correctness and practical executability.
3. Runtime-free verifiable rewards. Because CLI structure is deterministic and ground-truth commands were already validated during construction, component-level reward signals can be computed without executing model outputs, supporting scalable reinforcement learning.
4. Fine-grained evaluation and training gains. More than 20 LLMs are evaluated under three settings; argument construction is identified as the primary bottleneck, and SFT plus RLVR with KaliBench rewards improves an 8B model's mean Total Score across the three evaluation modes by 7.5 percentage points.

Main Findings

• No open-weight model exceeds 42% exact-command accuracy without tool hints. Across 24 configurations of general-purpose and security-focused open-weight models and three evaluation modes, the strongest unrestricted Exact Correct score reported is GLM-5.2 at 41.3%, followed by DeepSeek-V3.2 at 33.0%.
• Tool selection and argument construction are separate problems. Moving from Unrestricted to Restricted (20 candidate tool names including the correct one) raises average Tool Accuracy from 72.0% to 95.2%, but yields only modest gains in Optional-Argument F1, Positional-Argument F1, and Exact Correct. Restricting candidates fixes which tool to use; argument construction remains the bottleneck.
• Tool documentation gives the largest gains. In Hinted mode, average Optional-Argument F1 rises from 45.1% to 87.8%, Positional-Argument F1 from 61.1% to 90.1%, Total Score from 59.4% to 90.8%, and Exact Correct from 22.3% to 73.1%, suggesting failures often stem from limited domain knowledge rather than intent understanding.
• Optional arguments are the hardest component. The metric hierarchy reported places optional-argument prediction as most challenging, followed by positional-argument prediction.
• Scale alone does not determine performance. Qwen3 (8B, thinking) outperforms larger models including Gemma-3-it (27B) and Llama-Primus-Ins (70B) on Average Total Score.
• Cybersecurity-tuned models do not consistently win. In the compact group, RedSage-Ins scores higher Total Score than non-thinking Qwen3 in Unrestricted (56.4% vs. 55.5%) and Restricted (69.3% vs. 68.2%), but Qwen3 surpasses it in Hinted mode (91.6% vs. 89.5%). Domain-tuned models help more when guidance is scarce; general-purpose models exploit explicit hints better.
• Reasoning-enabled variants help at the argument level. Qwen3 (thinking) improves Average from 71.8% to 73.8% over Qwen3 and reaches 94.0% Total Score and 80.9% Exact Correct in Hinted mode.
• Post-training closes much of the gap. Three RedSage-K variants — SFT, GRPO, and SFT+GRPO — trained on KaliBench reach Average scores of 77.4, 76.9, and 79.2 respectively. RedSage-K (SFT+GRPO) ranks third in Average among all evaluated open-weight models, 1.0 point below DeepSeek-V3.2 (685B total / 37B active) despite an 8B backbone.
• Different training methods have complementary strengths. RedSage-K (SFT) achieves the best compact-model Tool Accuracy across all modes and the largest Hinted-mode improvements; RedSage-K (GRPO) shows stronger Positional-Argument F1 and Total Score in Unrestricted and Restricted but is weaker in Hinted; the combined SFT+GRPO variant gives the best compact Unrestricted/Restricted Total Score and Exact Correct, though GRPO-based variants slightly reduce Hinted-mode Total Score relative to the original RedSage-Ins, with argument-construction errors and regressions on previously solved examples identified in a paired analysis.
• Proprietary and scaffolded systems remain far from saturation. In Unrestricted mode on the 5,000 test queries, GPT-5.6-Sol reaches 61.68% full-set Exact Correct (4,988 queries answered), Codex with GPT-5.5 reaches 51.68% (4,633 answered), and Claude Opus 5 reaches 44.02% (3,675 answered; 59.89% on answered queries, i.e., 73.50% of the test set).
• Cross-model spread narrows in Total Score but not in Exact Correct. The Total Score range across models narrows from 21.8 to 18.4 to 13.7 percentage points across Unrestricted, Restricted, and Hinted, while the Exact Correct range stays wide at 28.8, 33.8, and 28.3 percentage points.
• Difficulty varies by capability dimension. Dimensions such as gpu and crypto-stego score consistently higher, while 802-11 and wireless show lower minima, especially in Unrestricted and Restricted modes; hardware remains comparatively low in Hinted mode, attributed to longer or more complex tool descriptions.
• Tool-level difficulty tracks interface complexity. Tools with simple invocation patterns and well-defined defaults (e.g., truecrypt2john, jsql) are handled reliably, while those with irregular or uncommon options (e.g., nikto, hashrat) remain challenging.
• Models are fairly robust to paraphrase but sensitive to messy input. Measured as average Total Score change in percentage points relative to original queries across eight model configurations: paraphrase +0.61/+0.22/-0.09, informal -0.55/-0.48/-0.28, and messy -3.20/-1.40/-1.07 for Unrestricted/Restricted/Hinted, with Hinted showing the smallest decline under messy input.
• Four recurring failure patterns. Long–short option misuse (e.g., mixing --domain and -d), option hallucination (positional arguments treated as flags, e.g., st --client), hallucinated long options for tools that only have short flags, and erroneous option names from singular–plural confusion (e.g., --symlink vs. --symlinks).

Methodology in Plain English

The authors started from official Kali Linux documentation for 2,372 tools, each described in structured Markdown with long and short options and aliases, plus a metapackage taxonomy they used to label each tool with a capability and functional dimension. For each tool, Qwen3-Max generated 10–13 natural-language query–command pairs with structured annotations of the tool name, alias-aware optional flag–value pairs, and positional arguments, producing 27.7K pairs.

Because hallucination was severe — initial manual inspection found over 50% of generated commands contained hallucinated or misspelled options, such as --silent instead of --silence — the authors filtered aggressively. An LLM-as-Verifier checked whether every flag and argument was documented and whether the command actually implemented the query, removing 53.8% of examples in the first round and leaving 413 tools (17.4%) without valid pairs, which triggered iterative regeneration; the LLM-validated set contained 14.5K pairs after 47.65% of the initial data was removed. Each surviving command was then executed in a Docker container running a Kali-linux-everything image, recording stdout, stderr, exit codes, and timeouts, with rule-based filtering rejecting a further 11.95%. A human-in-the-loop step, using an AI agent (OpenAI Codex with GPT-5.4 Thinking) to produce initial verdicts cross-checked by human reviewers, found an additional 4.9% invalid. Records were then deduplicated semantically with the SemHash framework — embedding [query, tool_name, ground_truth_command], clustering via approximate nearest-neighbor search, and grouping pairs with cosine similarity above 0.90 — yielding 8.5K pairs, 30.7% of the original generation.

Evaluation prompts models under three settings: Unrestricted (query only), Restricted (query plus 20 candidate tool names including the correct one), and Hinted (query plus official documentation of the candidate tools). Outputs are parsed with Python's shlex library rather than executed, and scored component-wise against recorded ground truth: Tool Accuracy, alias-aware Optional-Argument F1, multiset-matched Positional-Argument F1, Total Score (mean of tool and argument scores), and Exact Correct. Training used the same prompt template, concatenating all three settings for 3,504 × 3 = 10,512 samples, with LoRA and Unsloth on a single H200 GPU (141GB VRAM), total batch size 32, for two epochs, with RedSage-Ins as the backbone.

Why This Matters

The paper argues that general cybersecurity knowledge does not translate into operational competence, and that evaluating tool use at the level of command structure — tool name, optional flags, positional arguments — makes failure attribution possible in a way end-to-end agentic benchmarks cannot. Because KaliBench records deterministic, verified command components, it doubles as a source of runtime-free reward signals, making reinforcement learning practical for settings where executing model outputs would be unsafe or expensive.

Real-world applications the framing points to:

• Security operations assistance: turning analyst intent ("scan this subnet for SMB misconfigurations") into precise invocations of tools such as Nmap, avoiding the syntax errors that invalidate execution.
• Offensive tooling in controlled settings: structuring exploitation or scanning commands correctly (e.g., Metasploit) where argument-order and flag errors abort a run.
• Digital forensics and incident response: generating correct invocations of memory and artifact analysis tools such as Volatility under time pressure.
• Private deployment of security models: the paper emphasizes local deployment for sensitive queries, target information, and operational data that cannot be shared with proprietary APIs — reinforced by the finding that Claude Opus 5 answered only 73.50% of the test set, which the authors attribute to provider-specific safety restrictions.

Industry relevance: the near-absence of an open-weight model above 42% Exact Correct without tool hints, the 61.68% ceiling for the best proprietary system, and the observation that an 8B model trained with KaliBench rewards can approach a 685B MoE model all suggest that fine-grained, verifiable data and rewards may matter more for exact command generation than raw model scale.

Future Directions

• Multi-step workflows. The authors explicitly name this as a limitation-driven extension, since the current benchmark is limited to single-turn command generation.
• Retrieval-augmented tool use. Connecting models to documentation retrieval is proposed as a way to reduce reliance on memorized interface knowledge, given how much Hinted mode improves results.
• Environment-aware evaluation. Moving beyond documentation-grounded data toward settings that account for the runtime environment.
• Reducing Hinted-mode regressions from RL. The paired analysis shows that GRPO and SFT+GRPO variants slightly reduce Hinted-mode Total Score relative to RedSage-Ins, so understanding and mitigating argument-construction errors and regressions on previously solved examples remains open.

Target Audience

Researchers and engineers working on LLM tool use, function calling, and agentic evaluation; cybersecurity practitioners and red/blue team tooling developers interested in automating command generation; and teams building or post-training security-focused models who need a verifiable benchmark and reward source. The paper is also relevant to evaluation-methodology readers interested in schema-free, alias-aware, component-level scoring rather than end-to-end task success.

Authors’ abstract

LLMs are increasingly applied to cybersecurity workflows, where they are expected to translate analysts' intent into tool invocations. However, existing evaluations focus on knowledge-based assessments or end-to-end agentic tasks, and do not directly measure LLMs' ability to generate executable commands for real-world cybersecurity tools. This gap is critical because cybersecurity operations rely on strict command-line interfaces (CLIs), where minor syntax errors, incorrect flag--value bindings, or argument misordering can invalidate execution. We introduce KaliBench, a fine-grained benchmark and dataset for natural-language--to--CLI translation on Kali Linux, comprising 8,504 query--command pairs spanning 1,642 tools across 23 capability dimensions and 5 security phases. KaliBench is constructed via a manuscript-grounded pipeline with deterministic canonicalization and alias-aware evaluation, enabling precise and reproducible assessment of tool selection and argument construction. To ensure both semantic correctness and practical executability, we develop a multi-stage verification pipeline that combines LLM-based validation, sandboxed terminal execution, and human-in-the-loop refinement. Building on these fine-grained, deterministic signals, KaliBench further enables runtime-free verifiable rewards for training. Across three evaluation modes and 24 configurations of general-purpose and security-focused open-weight models, no open-weight model exceeds 42% exact-command accuracy in the unrestricted setting, highlighting the difficulty of accurate CLI-based cybersecurity tool use without explicit tool hints. We further show that supervised fine-tuning and reinforcement learning with verifiable rewards derived from KaliBench significantly improve an 8B model and achieve performance comparable to a 685B MoE model.