Intern-Decision: Efficient Multimodal Decision Making with Autoregressive Masked Language Modeling

AI summary

Overview

Research area: Natural Language Processing / multimodal agent systems — specifically compact "decision models" that return structured choices (rather than free-form text) from text plus optional images.

Technical level: Advanced. The paper assumes familiarity with causal language models, logits and softmax readout, cross-entropy objectives, and calibration metrics (Brier score, expected calibration error, temperature scaling).

Scope: The paper introduces Intern-Decision, a family of 0.8B, 2B, and 4B parameter multimodal decision models trained with an autoregressive masked language modeling objective, and reports decision accuracy on seven suites, single-request latency on one RTX 4090, probability calibration on a 96-case pilot, and a confidence-based handoff study on 111 Jevbench-Hard items.

What This Paper Is About

Many agent operations are closed-set: pick a tool, pick a route, pick a label, pick an action. Current agents still answer these with free-form autoregressive text, which adds decoding cost and makes the model-to-environment interface messy. The paper's goal is a compact model that reads a state (text plus optional images) and a typed question set, then in one forward pass returns a label plus a probability for every requested field — with reliable confidence so a bigger model can be called only when needed.

Key Contributions

1. Intern-Decision model family. 0.8B, 2B, and 4B multimodal decision models built on Qwen3.5 backbones, with the language backbone fine-tuned and the vision tower and projector frozen.
2. Autoregressive masked language modeling objective. Each decision field gets one <decision> placeholder inside a complete assistant JSON skeleton; ground-truth symbols appear only in supervision, and all fields are scored in a single causal forward pass using the backbone's native language-model head (readout from the position immediately before each marker, z_{p_j-1}).
3. A unified multimodal decision corpus. 17,880 training samples and 23,500 decisions, including 5,500 image-bearing samples, shared identically across the three released checkpoints (verified against input hashes, training logs, and tokenizer caches), mixing synthetic text decisions, public task distributions, and synthetic visual/interactive tasks.
4. Separated evaluation of accuracy, latency, and probability quality, plus a confidence-based handoff study and qualitative game, visual-selection, and browser demonstrations.

Main Findings

• Seven-suite accuracy: Intern-Decision-4B reaches a 90.02% unweighted average, versus 88.74% for Jev (v1.13.0), 85.16% for JevK5, 84.23% for SemIf, 79.56% for Kev, and 57.77% for Laya. The 4B model leads or ties on five of the seven suites.
• Per-suite 4B results: Easy 100.00%, Original 98.61%, Hard 73.87%, Typed Decision 80.55% (1,611/2,000 fields), ToolACE 96.45% (299/310), AG News 90.82%, WildJailBreak 89.86%. On public Jevbench it answers 48/48 Easy, 71/72 Original, and 82/111 Hard items.
• Where baselines win: Jev leads WildJailBreak (96.29%), Laya leads AG News (92.84%), and JevK5 ties the 4B model on Hard (73.87%).
• Size scaling within the family: the suite average rises from 79.38% (0.8B) to 84.68% (2B) to 90.02% (4B); the authors caution this cannot be attributed solely to parameter count without matched-data ablations or training-compute disclosures.
• Latency on a fixed 289-token, three-field request (one RTX 4090): Intern-Decision-4B mean 44.16 ms (P50 44.03, P95 44.60); 0.8B mean 33.98 ms; 2B mean 33.28 ms; Jev mean 109.70 ms. Mean ratios are 3.23×, 3.30×, and 2.48× respectively. Derived serial request rates are approximately 29.43, 30.05, and 22.64 per second — explicitly not measured saturated serving throughput.
• Hard-item probability metrics (111 items): the 4B model has the lowest Brier in the comparison (0.346762), while JevK5 has the lowest ECE (0.046672). Temperature scaling leaves the 4B's 82/111 correct decisions unchanged while reducing ECE from 0.132613 to 0.065267; Jev answers 80/111 correctly with ECE 0.094685.
• Known-distribution calibration pilot (96 cases): temperature scaling (T = 1.992418) reduces overall pooled expected Brier from 0.628 to 0.550 and expected ECE from 0.213 to 0.089, with all 96/96 predictions valid. Jev's pooled values are 0.595 and 0.130. The calibrated model beats its uncalibrated version in both metrics in all six categories, and beats Jev on Brier in five categories and on ECE in five.
• A case where calibration hurts: on the informed-host three-door item, the calibrated model returns (0.4222, 0.5778) with total variation distance 0.0889 from the reference, while the uncalibrated distribution (0.3486, 0.6514) is closer and Jev returns (0.18, 0.82) with TVD 0.1533 — so the aggregate gain is not uniform per case.
• Confidence-based handoff: routing 42 of 111 Hard items (37.84%, local coverage 62.16%) to Atria-Dawn-Preview raises accuracy from 82/111 (73.87%) to 97/111 (87.39%), 15 more correct than the small model and two fewer than Atria alone (99/111, 89.19%). Estimated hybrid total time is 140.16 s versus 300.03 s measured for Atria-only, a 53.29% reduction; the 4B-only 4.90 s figure is derived, not a per-item trace.
• Qualitative interaction demos: five game demonstrations (Snake, fruit catching, Mario, two first-person Doom tasks); four CAPTCHA-style visual selection trajectories; three browser trajectories reaching the requested evidence in 6, 6, and 4 local decisions with median inference latency of 101.6, 109.8, and 95.3 ms (server-reported, excluding planning, network, and browser execution).
• Interface limits: 62 candidate symbols supported; the HTTP service permits up to 16 fields per request — stated as interface limits, not generalization claims.

Methodology in Plain English

Instead of asking the model to write out an answer, the researchers gave it a form to fill in. The user turn contains the state (a message, optional images) and a schema of typed questions — each with a field name, an instruction, and an ordered list of candidate labels. The assistant turn is a JSON skeleton where every value is a single reserved placeholder token.

Each candidate label is mapped to one legal single-token symbol (so the model never has to spell anything). Training only supervises the symbol position for each field, using ordinary next-token cross-entropy against the full vocabulary; punctuation, state text, and schema descriptions are context, not targets. Because the backbone is causal, all fields are scored in one pass without feeding earlier answers forward.

At inference, the model looks only at the logits for legal candidates for that field, applies a softmax at a fixed temperature, and returns both the argmax label and the full probability distribution. The system therefore always emits schema-conforming output without a free-text decoder, though the authors note this guarantees format, not semantic correctness.

Two design choices matter for the results. First, a post-hoc temperature is fitted on a separate calibration split, which reshapes probabilities without changing any argmax. Second, because the model yields a maximum-candidate confidence, a simple cascade can keep the local answer when confidence is high and escalate to a larger model otherwise.

The data mixes teacher-generated hard and general text decisions (from deepseek-v4.1-flash: 4,066 General, 2,645 Hard), public task distributions (AG News, ToolACE, Typed Decisions, WildGuardMix, WildJailbreak, Ruozhiba), and synthetic visual tasks (rendered game frames, VizDoom, shapes/pointer actions). Evaluation is reported as accuracy per decision rather than per row, and the seven-suite average is an unweighted mean of suite accuracies, not pooled accuracy over the 12,351 decisions.

Why This Matters

Research impact. The paper argues that structured decision-making deserves its own model class rather than being a byproduct of free-text generation. It also contributes a reusable evaluation pattern: an exact-reference calibration test set (24 mechanism families, two settings each, canonical and reversed option orders, 48 paired settings) generated by a deterministic solver rather than Monte Carlo estimates or model outputs, with the reference distribution kept entirely out of the model's input.

Real-world applications:

• Tool and model routing in agent stacks, where a local model picks from a fixed action set and returns a probability a controller can threshold.
• Support triage, the paper's worked example: assigning a team, an urgency score, and a yes/no refund flag for one ticket in a single pass.
• GUI and browser automation, where a planner decomposes goals and a compact model repeatedly picks click/scroll/return/stop from live screenshots.
• Game and robot control, where screenshots feed discrete action choices without converting every frame into a symbolic description.
• Moderation and safety classification, represented by WildJailbreak and WildGuardMix in the training mixture.

Industry relevance. The 44.16 ms mean on a fixed short request, versus 109.70 ms for Jev in the same local protocol, points at latency budgets where free-form decoding is too slow. The handoff result is the commercially interesting one: routing 42 of 111 items to a bigger model captured most of that model's accuracy at a fraction of its measured time. The authors are explicit that the ratios need matched serving conditions and that batch size, warm-up, repetition count, and timing boundaries are unreported.

Future Directions

• Matched-data ablations. The paper repeatedly flags that the size trend and the benefit of joint field scoring are not isolated, because there are no matched-data ablations or training-compute disclosures. Separating these effects is the obvious next experiment.
• Better routing policies. Only one escalation operating point is reported (42 of 111 items routed at a threshold), described as illustrative rather than an optimized policy. Learned thresholds, cost-aware cascades, and multi-tier handoffs are open.
• Scaling the schema and the environment. The release supports 62 candidate symbols and up to 16 fields per request; generalization to large schemas, longer contexts, and genuinely dynamic environments is called an open problem. The latency figures come from one fixed 289-token request, not a sweep over long contexts or screenshots.
• Calibration beyond the pilot. The 96-case pilot has 48 paired settings rather than 96 independent trials, and no check establishes absence of semantic overlap with every training source. Broader, independently constructed distribution-matching tests — and cases where temperature scaling hurts, like the three-door item — need more study. Independence from verbose benchmarks is also unresolved: public Jevbench has informed development and is described as a diagnostic, not held-out evidence.

Target Audience

Engineers building agent systems who need fast, structured, confidence-bearing outputs from a small model; researchers working on decision models, model cascades, and calibration; and practitioners deploying multimodal models on constrained hardware. Readers wanting quantitative game-playing or browser-agent results will find only qualitative demonstrations here, and readers needing a controlled hardware comparison should note the paper's own caveats about unmatched serving conditions.

Authors’ abstract

Interactive agents repeatedly choose among a finite set of actions, labels, or ordinal outcomes. Generating a free-form response for each choice adds latency and complicates the interface between a model and its environment. We present Intern-Decision, a family of 0.8B, 2B, and 4B multimodal decision models that return structured choices and candidate probabilities from text and optional images. Our autoregressive masked language modeling objective places one decision placeholder per field in a complete assistant JSON skeleton. Ground-truth symbols appear only in supervision, while all fields are evaluated in a single causal forward pass using the backbone's native language-model head. The 4B model achieves a 90.02% unweighted average accuracy across seven evaluation suites, compared with 88.74% for Jev v1.13.0. For a fixed 289-token request with three fields on one RTX 4090, mean latency is 44.16 ms for the 4B model and approximately 33--34 ms for the smaller models. On a separate 96-case known-distribution pilot, temperature scaling reduces expected multiclass Brier from 0.628 to 0.550 and expected calibration error from 0.213 to 0.089. A confidence-based handoff study on 111 public Jevbench-Hard items improves accuracy from 82/111 (73.87%) to 97/111 (87.39%) by routing 42 items to Atria-Dawn-Preview. We also describe qualitative game, visual interface, and browser demonstrations. These results support compact decision models as a practical component of larger agent systems.