Quantigence: A Multi-Agent Framework for Post-Quantum Security Analysis on Commodity Hardware

AI summary

Overview

• Research area: Coding agents (LLM-based software agents) applied to post-quantum cryptography migration, with emphasis on interoperability testing rather than local self-testing.
• Technical level: Advanced. The paper assumes familiarity with ML-DSA (FIPS 204), key encoding formats (PEM/PKCS#8/SubjectPublicKeyInfo), cryptographic signing contexts, and LLM agent serving configurations.
• Scope (one sentence): The paper defines a contract-based Go file-signer migration from RSA-2048 to ML-DSA-44 and measures whether giving coding agents access to an independent interoperability checker improves their success across four local-agent configurations (160 attempts) plus four native-agent trials.

Note on the record: the metadata title supplied for this item ("Quantigence: A Multi-Agent Framework for Post-Quantum Security Analysis on Commodity Hardware") does not match the title printed in the paper content, which is "Can Coding Agents Migrate to Post-Quantum Cryptography?" by Abdulmalik Alquwayfili (Saudi Data & AI Authority). The summary below follows the paper content.

What This Paper Is About

A program can be migrated to post-quantum cryptography and still be broken: it can sign and verify its own signatures locally while producing keys or signatures that an independent implementation, such as OpenSSL, rejects. The paper builds a concrete migration task in which a Go file signer must move from RSA-2048 to ML-DSA-44 under a written contract, then asks whether coding agents do better when they can consult structured diagnostics from an external checker. The central question is whether such diagnostics help agents beyond simply having the verification tool available.

Key Contributions

1. A reproducible migration task and checker. A three-entry-point file signer (keygen, sign, verify) must preserve its interface while switching from RSA-2048 with PKCS#1 v1.5 (over a SHA-256 digest) to Pure ML-DSA-44, with persisted PEM-wrapped SubjectPublicKeyInfo public keys and PEM-wrapped PKCS#8 private keys, the context filesigner/v1, and distinct responses for invalid signatures (exit code 1) and malformed input (exit code 3).
2. Four paired comparisons of optional checker feedback. Each of four local-agent configurations runs 20 paired baseline–feedback attempts on matched seeds, holding the model, harness, tools, and editing budget fixed, so that checker access is the only difference.
3. A serving ablation that changes only context window. The Qwen3.8-27B configuration is re-run with the served context window reduced from 131,072 tokens to 32,768 tokens, with weights, draft head, inference engine, sampling, and paired seeds unchanged.
4. Recorded traces of incomplete repairs. The paper documents specific attempts (Qwen3-Coder baseline seed 5, feedback seed 4, feedback seed 7, feedback seed 1) showing what agents did after receiving diagnostics, including cases where a diagnostic was received but the underlying defect was not repaired.

Main Findings

• Checker access did not raise completion in any comparison. Across the four local comparisons, no condition with checker feedback showed a higher observed full-pass rate than its baseline.
• Twelve patches passed local verification but failed external requirements. Eight of these came from Qwen3-Coder (four local passes in each condition), the 32K Qwen3.8 batch added one, and Nemotron added three, for a total of twelve across the local configurations.
• Qwen3-Coder-30B-A3B full passes: 0/20 in both conditions. Builds were 13/20 baseline and 17/20 feedback, while local passes were 4/20 in each; all four local passes in each condition had at least one external-compatibility or required-rejection failure. The 95% Wilson interval was 0–16.1%.
• Qwen3.8-27B at 128K full passes: 18/20 in both conditions (95% Wilson interval 69.9–97.2%), with zero difference in percentage points. Paired outcomes: 16 pairs passed in both, two passed only in the baseline, and two passed only with feedback.
• Context window dominated this Qwen3.8 configuration. Reducing only the served window from 128K to 32K lowered full passes from 36/40 to 4/40 across the two conditions. At 32K the baseline passed 3/20 (interval 5.2–36.0%) and feedback passed 1/20 (interval 0.9–23.6%), an observed difference of −10 percentage points.
• At 32K, 21 attempts stopped on context errors and 19 on repeated unparseable responses. Nine of the latter ended with three consecutive responses that reached their output limit with reasoning but no command. None submitted voluntarily, compared with 27/40 at 128K. Six patches compiled at 32K versus 38 at 128K.
• Nemotron 3.5 Lightning full passes: 0/20 in both conditions at 128K. Twenty-eight attempts exhausted the 60-request budget and twelve stopped on repeated unparseable responses; the largest recorded prompt was 78,455 tokens, below the 131,072-token window. Six patches compiled and passed 12–25 checks; the three local passes all failed external requirements.
• Qwen3-Coder key handling blocked interoperability early. OpenSSL could not load the public key from any of the 30 built patches and loaded the private key from only one feedback patch. Both directions involving independently generated OpenSSL keys also failed for every built patch.
• Ten failed builds left 36 subsequent checks unassessed. Four baseline patches and one feedback patch stopped at compiler errors; two patches in each condition imported CIRCL components whose transitive dependencies were absent from the offline proxy; one further baseline patch deleted go.sum and added a replacement path into a temporary module cache.
• Qwen3.8 at 128K rarely needed the checker. Eighteen feedback attempts invoked it (twelve twice, six once), and every first call already passed all 40 checks. No checker-diagnosed repair was observed in that batch.
• Fourth comparison (Nemotron) rarely used the checker. Only two feedback attempts invoked it, once each, and both checks failed. The median attempt executed 56 actions and took 5.3 minutes.
• All four native-agent trials passed all 40 checks, including both baselines. Codex CLI 0.154.0 with gpt-6-astra used 12 actions and 6.5 minutes (baseline) and 10 actions and 5.6 minutes (feedback); Claude Code 2.1.272 with claude-fable-5-1 used 8 actions and 2.8 minutes (baseline) and 8 actions and 2.9 minutes (feedback).
• Codex's first development check failed 19 of 40 checks; its second and final evaluations passed all 40. Claude's single development check already passed all 40.
• Repair traces show diagnostics without repair. Qwen3-Coder baseline seed 5 and feedback seed 4 both wrote packed CIRCL key bytes inside PEM blocks; local signing and verification passed, OpenSSL rejected both keys, and each final evaluation recorded 22 passes and 18 failures. Feedback seed 7 fixed a context mismatch by switching to SignTo, and still recorded 22 passes and 18 failures. Feedback seed 1 received a key-generation diagnostic but retained the faulty binding and marshaling call, recording 9 passes, 30 failures, and one unassessed signature-size row.
• Costs were similar between conditions. For Qwen3-Coder, median attempt time was 27.6 minutes baseline versus 29.5 minutes with feedback; for Qwen3.8 at 128K, median executed actions were 54.0 versus 45.5 but median elapsed time was 25.7 versus 26.1 minutes.
• Reference and validation controls behaved as intended. The v4 reference passes all 40 checks, and the evaluator detected all 16 validation controls, including through the isolated scoring path. These are excluded from agent results.

Methodology in Plain English

The researchers wrote a specification ("contract") for migrating a small Go command-line file signer from RSA to ML-DSA-44 and gave it to several coding agents running in a controlled sandbox. Every agent got the same source code, documentation, compiler, and an OpenSSL installation that supports ML-DSA-44, and the same budget.

Each agent was tested in two conditions on matched random seeds: a baseline that could write its own tests and call OpenSSL directly, and a feedback condition that additionally had an optional check command returning named pass/fail diagnostics from an independent checker. The checker runs the candidate program as a subprocess and exchanges keys and signatures with OpenSSL in both directions, plus rejection tests (altered messages, wrong keys, changed or removed context, corrupted and truncated signatures, unparseable keys) and persistence checks. After the agent stopped, a separate evaluator rebuilt the final patch on fresh inputs and scored all 40 checks, with no repair feedback. Outcomes counted local round-trip passes separately from full external passes, using counts, 95% Wilson intervals, differences in percentage points, and paired-seed categories.

Four local configurations were compared, all on mini-swe-agent 2.4.6 with LiteLLM 1.100.1: Qwen3-Coder-30B-A3B-Instruct via vLLM 0.24.0 at 32,768 tokens; Qwen3.8-27B in Q4_K_M via llama.cpp revision 4a89937 at 131,072 tokens; the same Qwen3.8 setup at 32,768 tokens; and Nemotron 3.5 Lightning (30B parameters, 3B active) in Q4_0 at 131,072 tokens. Four additional single trials used native CLI agents (Codex and Claude Code) and were reported separately rather than as controlled comparisons.

Why This Matters

Impact on research. The paper separates two things that are usually conflated: a program agreeing with itself and a program interoperating with an independent implementation. It also shows that serving settings, in this case context window, can change measured completion more dramatically than the experimental intervention itself, arguing for reporting harness and serving limits alongside agent results.

Real-world applications (bullets):

• Migrating signing services in TLS, code-signing, and document-signing infrastructure from RSA to ML-DSA, where keys must load in other vendors' stacks.
• Evaluating coding-agent tools for cryptographic maintenance in regulated software, where "tests pass" is not the same as standards-compliance.
• Building interoperability and rejection test suites into migration pipelines, so that a serialization defect does not silently propagate into downstream checks.
• Choosing inference configurations (context window, quantization, serving engine) for agent deployments on commodity GPUs, since window size alone moved full passes from 36/40 to 4/40 in one configuration.

Industry relevance. Organizations adopting FIPS 204 and planning post-quantum transitions need to know whether agent assistance actually reduces migration risk. This paper's answer is cautionary: in 160 counted attempts, optional checker diagnostics never increased the observed completion rate, and twelve patches passed local verification while failing external requirements.

Future Directions

• Required versus optional checking. The study tests optional access to diagnostics, not a workflow that forces a check or a repair after each failed validation; the authors explicitly note this interface is untested.
• Diagnostics that expose dependency ordering. The paper suggests verifying key exchange before signature and rejection behavior so that one serialization error does not produce several dependent failures, but does not test such an interface.
• Broader task coverage. Extending beyond one Go file signer, one ML-DSA parameter set, and one external implementation (OpenSSL) remains open, as does integrating known-answer tests for FIPS conformance and checks for side-channel resistance.
• Separating model capability from harness and serving effects. The Qwen3-Coder failure was not isolated (its larger-window version was not tested), and cross-model comparisons also changed precision, inference engine, context, sampling, and load; controlled estimates would require holding those constant.

Target Audience

Researchers and practitioners in applied cryptography migration, LLM coding-agent evaluation, and secure software engineering; engineers planning RSA-to-ML-DSA transitions in signing services; and teams building agent benchmarks who need to distinguish local test agreement from external interoperability and to report serving and harness limits alongside agent outcomes.

Authors’ abstract

The migration to post-quantum cryptography (PQC) forces security teams to synthesize a fast-moving literature spanning lattice theory, implementation security, and shifting NIST policy. We present Quantigence, a multi-agent framework that structures this analysis as a supervisor that decomposes a query into sub-tasks and dispatches them to four specialist agents (cryptographic analysis, threat modeling, standards compliance, and risk assessment), each grounded in live external tools: the arXiv and NVD APIs and a local retrieval index over the NIST PQC standards. Agents run serially on a single commodity GPU using a 4-bit-quantized 9B model, keeping the peak memory footprint within an 8 GB budget. The framework computes a Quantum-Adjusted Risk Score (QARS), an automated operationalization of the risk model of Grigaliūnas and Brūzgienė, to prioritize assets. On a 40-query benchmark of atomic, machine-checkable questions, tool access is what raises accuracy, from 57% (zero-shot) to 98% (single agent), and a single tool-using agent is, as expected, a strong baseline that multi-agent decomposition does not beat on single-fact lookups. On a second benchmark of ten complex, multi-faceted queries (the setting orchestration is designed for), the supervisor/worker decomposition instead improves rubric coverage from 78% to 89% and judged answer quality. We report both, and further measure robustness to retrieval-corpus poisoning and behavior across model scales. The implementation and benchmark are released as open source.