DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision

AI summary

Overview

Research area: Robustness and security evaluation of decision models (JEV-style models that map instructions and observations directly to choices, binary judgments, and scores), spanning both text and vision inputs.

Technical level: Advanced. The paper assumes familiarity with adversarial perturbations, prompt injection, attack success rate (ASR) metrics, and transfer-based evaluation. The benchmark itself is usable by practitioners with less background.

Scope: The paper introduces DecisionRobust, a benchmark that measures how 30 text model configurations and 17 vision model configurations behave under character-, word-, sentence-, sensor-, and image-level perturbations plus prompt injection across application-oriented decision tasks.

What This Paper Is About

Decision models such as JEV take an instruction, an observation, and a set of options or scoring criteria at inference time, then return a choice, a judgment, or a score rather than free-form text. Because these models are used in automated workflows, a response can satisfy the output schema while silently selecting the wrong action, so clean-input accuracy alone does not establish reliability.

The paper's goal is to build and run a curated benchmark that exposes how such models fail under attacker-controlled text, images, and injected instructions, and to verify that apparent failures are genuine rather than artifacts of invalid edits or lost task evidence.

Key Contributions

1. Application-grounded coverage: Decision tasks drawn from practical use cases — content filtering, tool routing, workflow assessment, retrieval, GUI targeting, physical state and motion, market forecasting, visual decisions, and language diagnostics — with native Choice, Noul, and Score interfaces preserved and tested under both textual and visual threats.

2. A curated evaluation dataset: Attack candidates filtered through external label-consistency checks, semantic review of text edits, and visual-readability assessment, then selected for cross-model evaluation via transfer-based selection.

3. A systematic evaluation: Analysis of adversarial robustness and transfer across models, including 24 held-out text models and 12 held-out vision models that were excluded from candidate selection, so that vulnerabilities are shown beyond the construction and selection models.

4. Metrics and validity machinery: Sample-level and candidate-level ASR definitions, plus flip ASR and target ASR for injection, together with a released full version containing all candidates that passed the applicable label-consistency checks before selection.

Main Findings

• Attack transfer beyond selection models: All 24 held-out text models incur errors on the curated text sets, showing vulnerabilities that extend past the models used for candidate selection. The image sets likewise transfer beyond the five vision selection models, though image perturbation ASR is low on several held-out models.

• Optimized image attacks transfer unevenly: Despite optimizing against three surrogates (Imajev-4B, Visual-Jev-4B, Jev-Omni), label-consistent ASR stays below 10% on seven of the ten transfer targets in Appendix A.6. On the 160-pair optimized set, OneJev-4B is most susceptible (39.8 raw / 32.4 checked) while StandardOne-8B shows 7.0 / 6.2 and StandardOne-3B shows 10.9 / 8.9.

• Perturbation resistance does not imply injection resistance: GLiNER2.5-Decide-340M has the lowest perturbation ASR (11.4%) but 55.3% injection target ASR. On vision, Perplexity Decider v1.1-27B has the lowest perturbation ASR (3.2%) yet 45.7% instruction-injection target ASR.

• Label-consistency checks substantially reduce measured ASR: On text perturbations across all datasets and formats (N = 1332), Combined ASR falls from 69.7 raw to 36.6 checked on Jev, from 49.1 to 20.0 on KEV, from 48.8 to 12.5 on AutoTrust, from 45.4 to 12.8 on OpenJEV, from 38.9 to 22.5 on Bosun, and from 51.9 to 19.5 on Liquid D1. Many raw successes on NevIR and Siim fail the checks.

• Robustness varies by task, not just by model: StandardOne-8B has the highest checked ASR on four common-perturbation datasets but the lowest on FinQA. OpenJEV has the highest checked ASR among transfer targets on PromptShield Noul but the lowest on MMLU Noul.

• No single injection template dominates: Repetition is strongest on KEV and Bosun at both append locations and under both metrics. Jev and OpenJEV are most affected by Self-check in observations but Direct in instructions, so both wording and location matter.

• Injection location effects are model-dependent: Observation injection yields higher flip and target ASR on Jev, KEV, AutoTrust, and Bosun, while OpenJEV and Liquid D1 show the reverse across the combined templates. Target ASR is always below flip ASR, meaning a wrong decision need not reach the attacker's chosen answer — the gap is especially pronounced on Score tasks.

• In-image injection still works after validation: Label checks reduce in-image ASR on all five vision selection models, but injection still redirects decisions when visual evidence supports the reference. Visual-Jev-4B has the highest checked in-image target ASR (79.4), Jev-Omni is most susceptible to instruction injection (93.0 flip / 93.0 target), and in-image ASR exceeds instruction ASR on four of five models.

• Search cost and transfer do not track each other: Charmer has the highest ASR on every text model but its advantage shrinks substantially in transfer, while SCPN and StressTest have higher ASR on Bosun than on Jev. Charmer also has the highest mean search cost at 5,755.7 queries per clean-correct sample versus 11.2 for SCPN.

• Attack suite beats ordinary variation: On Jev, combined attacks exceed unchanged-input reruns (Repeat@8) in raw ASR and paraphrasing (Paraphrase@4) in checked ASR across all 16 text source groups.

• Defocus blur is the most disruptive common image perturbation after checks, and resolution reduction ranks second by raw ASR but falls below Gaussian noise on Jev-Omni and StandardOne-8B after label checks. Combining perturbation types raises label-consistent ASR on all five models.

Methodology in Plain English

The researchers surveyed official examples, public GitHub projects, and community applications to pick decision tasks that are actually used in practice, then kept each task's native interface: Choice (pick an option), Noul (judge a proposition), or Score (rate on a supplied scale). They adapted screenshot grounding into element selection and rendered FinQA tables as images, and added SST-2 and MMLU as general language diagnostics.

For attacks, they defined a threat model with different levels of attacker control: text edits that must preserve task-relevant meaning, sensor attacks on permitted numerical and categorical fields, image pixel changes, and prompt injection that appends attacker instructions to observations, question instructions, or images. Text perturbations were constructed against Jev (jev-1.13.0) using DeepWordBug, Charmer, TextFooler, SememePSO, SCPN, PromptAttack S2/S3, and StressTest. Image attacks used five common corruptions at five severities plus optimized perturbations generated against three surrogate models with momentum updates, random resizing and padding, 200 steps, an L-infinity budget of 8/255, and pixel clipping to [0,1].

Before evaluation, they validated attacks. Two external LLMs independently answer the perturbed task without seeing the reference and both must recover it — GPT-5 mini with DeepSeek-V4-Flash for text, Claude Sonnet 5 for images; in-image injection additionally requires GPT-5 and Claude Sonnet 5 to recover the reference from readable visual evidence while ignoring injected text. Text edits also went through GPT-5 semantic review (facts, entities, quantities, relations, conditions, and intent must not change) plus human review of every example. Candidates that fooled more of the six text or five vision selection models were favored, balancing task coverage. All evaluation models outside those selection sets count as held-out.

Evaluation reports attack success at two levels: sample-level (a sample counts as a success if any of its candidates succeeds) and candidate-level (each variant counted individually). Injection adds flip ASR (any wrong decision) and target ASR (matching the predefined attacker answer). Score decisions are counted correct when the API probability-weighted mean is within 0.5 of the reference.

Why This Matters

Impact on research. The paper argues that prediction changes alone overstate attack effectiveness, and demonstrates this quantitatively — label checks cut measured ASR roughly in half on text and reduce it on every image dataset and model. It also shows that robustness is not a single property: resistance to perturbations and resistance to prompt injection come apart, as the GLiNER2.5-Decide-340M and Perplexity Decider v1.1-27B results illustrate. Benchmarks that evaluate only one threat, or only aggregate accuracy, will miss the others.

Real-world applications:

• Content filtering and moderation: identifying unsafe content, spam, or injected instructions, tested using AEGIS2.0-sourced tasks.
• Tool and skill routing: selecting a function or skill for a request, drawing on BFCL and Jev Agent Skill Router, where a meaning-preserving edit or an injected note can silently redirect the chosen tool.
• Retrieval and reranking: selecting relevant passages or judging product relevance, based on NevIR and ESCI, where retrieved content may itself be attacker-controlled.
• GUI agents and visual decisions: selecting interface elements from DOMs or screenshots (ScreenSpot, Multimodal-Mind2Web, jevnav) and applying counting, spatial, or conditional rules to rendered tables and geometry diagrams (FinQA, Geometry3K).

Industry relevance. Organizations deploying decision models inside automated workflows — routing, approval, filtering, ranking — need to know whether attacker-controlled input can flip a decision while the output still validates against the schema. The benchmark's curation pipeline (label consistency, semantic review, readable-evidence checks) is directly reusable as an acceptance-testing practice, and the release of a full unfiltered candidate set lets teams run the harder, unselected cases themselves.

Future Directions

• Stronger transferable image attacks. The paper states that optimizing against multiple surrogates does not ensure broad transfer under this attack setting, and calls for more transferable image attacks as future work.

• Extending beyond individual decisions. The limitations section notes the benchmark covers single decisions rather than their downstream effects in complete workflows, so measuring end-to-end workflow consequences is an open question.

• Broadening the attack suite. Computational constraints limited which attack methods and configurations could be evaluated and how widely they cover tasks, leaving room for larger and more varied suites.

• Understanding why robustness diverges. The findings that task-level robustness rankings shift (StandardOne-8B highest on four common-perturbation datasets, lowest on FinQA) and that perturbation and injection resistance come apart raise the question of what model or task properties drive each failure mode.

• Evaluating the unselected candidate pool. The announced full version, containing all candidates that passed applicable label-consistency checks before transfer-based selection and semantic or readability review, invites re-evaluation on harder or differently biased subsets.

Target Audience

Researchers and engineers working on model robustness, adversarial machine learning, and LLM security; teams building or auditing decision-model pipelines for routing, filtering, ranking, and workflow approval; benchmark designers interested in attack-validity methodology such as label-consistency checks and semantic review; and practitioners working on multimodal agents, GUI automation, or document-based visual question answering who need to know how their models behave when inputs are attacker-controlled.

Authors’ abstract

Emerging decision models such as JEV provide typed interfaces that map task instructions and observations directly to choices, judgments, and scores. As these models are increasingly used for automated decision-making, understanding their reliability under adversarial manipulation becomes critical. We introduce DecisionRobust, a benchmark for systematically evaluating the robustness of decision models across textual and visual inputs. Our benchmark covers practical decision-making scenarios and a diverse suite of attacks, including semantics-preserving text perturbations, image perturbations, and prompt injection in both modalities. We curate evaluation sets through external validity checks and cross-model candidate selection. We evaluate 30 model configurations on text tasks and 17 on vision tasks, including 24 and 12 held-out models, respectively, that are excluded from candidate selection. Our experiments reveal several key findings. First, attacks expose vulnerabilities beyond the models used for construction and selection, although optimized image perturbations transfer weakly to many targets even with multiple surrogates. Second, relative model robustness varies across tasks and attack types, and resistance to input perturbations does not necessarily imply resistance to prompt injection. Third, label-consistency checks substantially reduce measured attack success across text and image settings, highlighting the importance of validating apparent failures. The benchmark is available at https://yifan-lan.github.io/DecisionRobust/.