CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning

AI summary

Overview

Research area: Multimodal large language models (MLLMs), tool-integrated visual reasoning, reinforcement learning for agents. Technical level: Intermediate (assumes familiarity with chain-of-thought reasoning, supervised fine-tuning, and policy-gradient RL). Scope: The paper introduces CodeDance, a 7B multimodal agent that uses executable Python code as a unified medium to invoke, compose, and verify visual tools during reasoning, trained with SFT on a 34K trajectory dataset followed by RL with a difficulty-adaptive tool reward.

What This Paper Is About

Most open-source multimodal models reason about images using text-only chains of thought or rigid, predefined visual schemas (for example, only predicting bounding-box coordinates for cropping). This limits flexibility, interpretability, and transfer to new tasks, and it leaves unresolved the question of when a model should invoke a tool at all. CodeDance addresses this by letting the model write, compose, and execute code that orchestrates multiple visual tools, computes intermediate results, and renders artifacts such as boxes, lines, and plots — producing a transparent, self-checkable reasoning trace.

Key Contributions

1. CodeDance, an executable-code reasoning framework. A multimodal agent that "thinks with images" by planning and composing visual–symbolic operations through executable code as a unified medium, rather than fixed operation templates.
2. A 34K high-quality SFT dataset. Curated trajectories covering diverse atomic code capabilities including cropping, drawing, and point plotting, produced via weak-to-strong filtering (using Qwen2.5-VL-7B for difficulty stratification and Qwen2.5-VL-72B for cross-validation) and multi-turn atomic supervision.
3. A difficulty-adaptive reward mechanism (R_BAT) for RL. A two-level reward combining a sequence-level, group-accuracy-conditioned term and a turn-level execution term that encourages longer operation chains for hard problems and discourages unnecessary calls for easy ones.
4. Empirical documentation of emergent behaviors. Despite training only on atomic operations, the model spontaneously exhibits novel tool routines, unseen operation compositions, and cross-task transfer during RL, verified in ablation studies.

Main Findings

• Counting and chart performance. CodeDance-7B reaches 91.2 on CountBench, 77.1 on PixmoCount, and 87.5 on ChartQA, reported as state-of-the-art on Counting and ChartQA among the compared models. Relative to Qwen2.5-VL-7B the reported deltas are ↑19.2% (CountBench), ↑53.0% (PixmoCount), and ↑1.4% (ChartQA).
• Visual search and general benchmarks. CodeDance-7B scores 84.8 on V* Bench, 75.2 on HR-Bench 4K, 72.3 on HR-Bench 8K, and 44.1 on CharXiv, with deltas versus Qwen2.5-VL-7B of ↑11.0%, ↑9.0%, ↑9.5%, and ↑4.7% respectively. The paper states CodeDance-7B exceeds the 32B Qwen2.5-VL version on some tasks.
• Surpassing closed models. On V* Bench CodeDance-7B (84.8) exceeds GPT-4o (67.5), and on PixmoCount (77.1) it exceeds Qwen2.5-VL-72B (62.3).
• Math reasoning gains. CodeDance-7B improves MathVision from 25.0 (Qwen2.5-VL-7B) to 29.6 (+18.4%) and WeMath from 35.4 to 39.6 (+11.9%), while scoring 70.3 on MathVista and 46.8 on MathVerse.
• Reward design matters. In the ablation, SFT cold-start alone averages 62.6 accuracy at 1.71 turns; outcome-only GRPO reaches 66.8; the DeepEyes-style reward inflates turns to 2.26 with 65.6 accuracy; R_BAT without the turn-level reward reaches 67.5 with an execution success rate of 0.91; the full R_BAT achieves 68.1 accuracy at 1.38 turns with a 0.99 execution success rate.
• Turn-level reward prevents collapse. Without the turn-level reward, policy entropy collapses quickly; with it, entropy collapse is delayed and validation accuracy is consistently higher.
• Scaling across four dimensions. Enlarging SFT data from 5K to 34K yields steady gains; increasing model capacity from 3B to 7B substantially boosts counting and search (CodeDance-3B even outperforms a stronger 7B model on those benchmarks); extending RL training to 240 steps improves accuracy without overfitting; and raising the inference turn budget from 6 to 10 improves performance, with additional gains beyond 6 turns reported as 0.3%.
• Emergent cross-domain tool transfer. A bounding-box operation introduced for chart tasks in the SFT data was repurposed during RL for counting: the model localizes candidates with boxes, validates each localization, then derives the count.
• Emergent composition. Tasks restricted to a single predefined operation in SFT were solved by combining operations after post-training — for example, pointing to verify house position followed by crop with zoom-in to inspect fine details.
• Emergent unseen tools. The paper reports the model writing code not defined in the SFT data, such as OpenCV functions (for example cv2.rectangle to overlay a grid while counting headsets), attributed to pretrained knowledge activated during post-training.

Methodology in Plain English

The reasoning unit is a think–execute–feedback cycle. Given a query with text and an image, the policy model produces a trajectory interleaving natural-language reasoning with executable code. The code runs in an isolated sandbox with a dedicated working directory and namespace, disables certain APIs, enforces a 15-second per-call wall-clock limit, and returns structured stdout/stderr plus graphical outputs from libraries such as matplotlib and PIL as textual and visual observations. These observations are fed back into the next turn, and the loop continues until a final answer is produced or a maximum turn budget M is reached.

Training happens in two stages. First, supervised fine-tuning on the 34K curated dataset teaches atomic capabilities (image transforms such as crop and resize; mathematical computation such as measurement, algebra, and aggregation; open-ended visual editing such as drawing and annotation). Data construction uses a two-step pipeline: weak-to-strong filtering to remove trivial or low-information cases and stratify remaining data by difficulty, followed by multi-turn atomic supervision, with each trajectory verified by a stronger MLLM.

Second, reinforcement learning with GRPO refines behavior. The composite reward is r(τ) = R_acc + R_format + R_BAT. R_BAT decomposes into a sequence-level term R_seq and a turn-level term R_turn. R_seq scales the reward for successful tool calls by a factor d = σ(γ(0.5 − μ_acc)) − δ, where μ_acc is group-level accuracy: when most rollouts already succeed (high μ_acc), the factor shrinks to discourage redundant calls; when accuracy is low, it grows to encourage exploration. The authors set γ = 4 and δ = 0.2. R_turn assigns an immediate −0.5 penalty when code execution fails and 0 otherwise, then accumulates a discounted return with β = 0.2, normalized batch-wise to form A_turn. The final advantage is A(τ) = A_seq(R_acc, R_format, R_seq) + A_turn(R_turn).

Implementation uses Qwen2.5-VL-7B as the base model, SWIFT for SFT, VeRL for RL, and VLMEvalKit as the evaluation framework. SFT uses batch size 128, learning rate 5e-5, and warmup ratio 0.05; RL uses BF16 precision, global batch size 256, rollout of 8, 1 total epoch, and roughly 2 days of training. Maximum turns are set to 10 for evaluation and 6 for training. Deployment for inference and evaluation uses vLLM.

Why This Matters

• Research impact. The paper argues that executable code is a general, verifiable medium for multimodal reasoning, in contrast to fixed-schema pipelines and proprietary black-box systems like o3 whose internal mechanisms are inaccessible. It provides an open recipe (data pipeline, reward design, sandbox) and documents emergent generalization as evidence that the approach scales.
• Real-world applications (derived from the benchmarks studied):
  • Counting and object enumeration in dense or cluttered natural images.
  • Chart and document question answering, where values must be read accurately and compared.
  • High-resolution visual search, including locating small targets in 4K and 8K images.
  • Mathematical and symbolic reasoning over diagrams and figures.
• Industry relevance. For teams building multimodal agents, the work offers a reward-design template for balancing tool exploration against efficiency, an emphasis on avoiding tool spam and reward hacking, and a sandboxing approach (isolation, timeouts, atomic rollback, captured logs) that makes model-generated code practical to run during both training and deployment.

Future Directions

• Characterizing emergent behaviors. The paper documents novel tool invocations, unseen compositions, and cross-task transfer, but the mechanisms behind them — attributed to pretrained knowledge shaped by adaptive rewards — are described as empirically observed rather than fully explained.
• Scaling the recipe further. Figure 4 studies SFT dataset size, model capacity, RL steps, and inference turns; how far each axis can be pushed, and where returns diminish, is left open.
• Broadening tool and task coverage. The current study focuses on atomic image transforms, mathematical computation, and visual editing; extending to other tool types and task families is a natural next step.
• Reward hyperparameter sensitivity. The authors set γ = 4 and δ = 0.2 in the main experiments and defer discussion of these hyperparameters to the appendix, leaving the generalization of these settings across models and datasets as an open question.

Target Audience

Researchers and engineers working on multimodal agents, tool-augmented reasoning, and RL post-training of vision-language models. It is also relevant to practitioners who need transparent, verifiable intermediate evidence for visual tasks such as counting, chart QA, visual search, and diagram-based math, and to those studying emergent capabilities in reinforcement learning.

Authors’ abstract

Recent releases such as o3 highlight human-like "thinking with images" reasoning that combines tool use with stepwise verification, yet most open-source approaches still rely on text-only chains, rigid visual schemas, or single-step pipelines, limiting flexibility, interpretability, and transferability on complex tasks. We introduce CodeDance, which explores executable code as a general solver for visual reasoning. Unlike fixed-schema calls (e.g., only predicting bounding-box coordinates), CodeDance defines, composes, and executes code to orchestrate multiple tools, compute intermediate results, and render visual artifacts (e.g., boxes, lines, plots) that support transparent, self-checkable reasoning. To guide this process, we introduce a reward for balanced and adaptive tool calling, which balances exploration with efficiency and mitigates tool overuse. Interestingly, beyond the expected capabilities taught by atomic supervision, we empirically observe novel emergent behaviors during RL training: CodeDance demonstrates novel tool invocations, unseen compositions, and cross-task transfer. These behaviors arise without task-specific fine-tuning, suggesting a general and scalable mechanism for executable visual reasoning. Extensive experiments across reasoning benchmarks (e.g., visual search, math, chart QA) show that CodeDance not only consistently outperforms schema-driven and text-only baselines, but also surpasses closed models such as GPT-4o and larger open-source models.