Learning to Group and Order: Cross-Instance Self-Supervised RL for Vision-Centric MLLMs

AI summary

Overview

Research area: Computer Vision / multimodal large language models (MLLMs) — specifically self-supervised reinforcement learning (RL) post-training for vision-centric visual understanding.

Technical level: Intermediate. Readers should be comfortable with MLLM post-training, reinforcement learning with verifiable rewards (RLVR), and visual pretext tasks such as jigsaw solving; the paper's core idea is explained in plain terms.

Scope (one sentence): The paper introduces GO-SSL, a self-supervised RL framework that mixes local visual elements from two different instances and rewards a pretrained MLLM for both grouping elements by their source and recovering their order within each source, evaluated on image, 3D, and video benchmarks.

What This Paper Is About

Multimodal LLMs are good at general visual question answering but still struggle with fine-grained perception and spatial/relational reasoning. RL post-training with automatically verifiable rewards is a promising fix, and prior work (SSL4RL, Spatial-SSRL, Visual Jigsaw) builds such rewards from self-supervised pretext tasks like patch reordering — but these tasks only use within-instance structure, so the model learns where a patch belongs, not which image it came from. GO-SSL makes cross-instance comparison an explicit target: given two images (or two RGB-D scenes, or two videos), the model must split a shuffled pool of local elements back into the two source groups and simultaneously reconstruct the order inside each group.

Key Contributions

1. A cross-instance group-and-order framework (GO-SSL). The paper converts unlabeled visual data into verifiable "group + order" tasks by interleaving local elements from two sources and requiring both source membership and within-source ordering, making fine-grained instance discrimination an explicit optimization target rather than an implicit side effect.

2. Pairing as a source of supervision diversity. The authors formalize the idea that pairing instances yields many comparative contexts from the same unlabeled pool — single-source jigsaw offers roughly n contexts while pairwise GO-SSL permits the binomial count of source pairs (O(n²)) — and control source ambiguity through a random-to-hard curriculum (random pairs, then CLIP-Hard or CLIP-Blind pairs mined with CLIP and DINOv2 features). The paper is careful to note this combinatorial count is a property of task construction, not a measured sample-complexity or compute-efficiency guarantee.

3. A modality-agnostic instantiation across images, 3D, and video. The same formulation is applied to cross-image patch jigsaw (primary setting), cross-scene depth ordering on RGB-D crops, and cross-video temporal ordering.

4. Empirical evidence of gains and generality. GO-SSL beats the base MLLM and prior self-supervised post-training baselines on image, 3D, and video benchmarks, with additional checks across model scales and families (Qwen2.5-VL-3B and LLaVA-OneVision-7B) and targeted cross-instance discrimination tests.

Main Findings

• Image benchmarks (14 metrics, unweighted average): Image GO-SSL (CB-2×3) reaches an average of 60.90, versus 56.75 for base Qwen2.5-VL-7B (a +4.15 gain), 59.04 for Spatial-SSRL (+1.86), and 59.13 for Visual Jigsaw (+1.77). The simpler Image GO-SSL (CB-600) reaches 60.85.

• Where the image gains appear (CB-2×3 vs. base): HRBench4K +5.50 (74.00), HRBench8K +6.99 (72.37), V* +5.24 (82.20), MMStar +4.40 (75.20), VSR +2.37, SpatialEval-maze +15.00 (42.60), CVBench2D +1.83, DA-2K +6.91 (61.36), OmniSpatial +2.61, SugarCrepe++ +0.40, SeedBench +0.19, NaturalBench +1.74, Hallusion +1.16, EMMA +3.72.

• Modest training budget on the image side: Image GO-SSL uses 1000 steps with global batch size 64, compared with Visual Jigsaw's 1000 steps at global batch size 256 and Spatial-SSRL's 81,053 self-supervised QA pairs plus an SFT warm-up.

• 3D extension: 3D GO-SSL averages 57.31, a +8.04 gain over the base model's 49.27, and above Spatial-SSRL (52.28) and 3D Visual Jigsaw (51.78). Largest gains are DA-2K +20.45 (74.90) and BLINK +13.71 (91.13); also Omni3DBench +7.15 (24.29), 3DSRBench +0.93, Static-EmbodiedBench +2.50, OmniSpatial +3.52.

• Video extension (16-frame setting): Video GO-SSL averages 56.34, a +3.34 gain over the base model's 53.00 and +1.07 over Video Visual Jigsaw's 55.27. The largest gain is on AoTBench +10.23 (55.75), alongside TUNABench +2.31, VideoMME +1.40, TempCompass +1.52, VSI-Bench +2.74, Video-TT +2.40.

• Grouping and ordering are complementary: A group-only variant scores 57.14 and an order-only variant scores 58.43, both above the 56.75 baseline, while the combined random-pair G&O model reaches 60.19.

• Hard-pair construction matters, and CLIP-Blind beats CLIP-Hard: CLIP-Hard-600 reaches 60.43 and CLIP-Blind-600 reaches 60.85 under the same budget. The authors attribute this to CLIP-nearest pairs being similar mainly in global semantic space while still separable at patch level, whereas CLIP-Blind pairs are close under CLIP but separated under DINO, creating more informative ambiguity.

• Longer curriculum and finer granularity help unevenly: Moving from CLIP-Blind-400 (60.40) to CLIP-Blind-600 (60.85) improves the average; the denser 2×3 layout gives the best detail-understanding/visual-perception score and the best average (60.90) but does not uniformly improve every category.

• Decoy candidates do not pay off overall: GO-SSL-Decoy-Explicit scores 60.31 and GO-SSL-Decoy-Implicit 60.46, both below CLIP-Blind-600's 60.85, although decoys improve "extra" benchmark categories — the simpler formulation captures most of the benefit with less reward-design complexity.

• Explicit reasoning does not consistently help: In the answer-only versus chain-of-thought comparison across image, 3D, and video benchmarks with Qwen2.5-VL-3B and Qwen2.5-VL-7B, the paper reports that enabling CoT does not consistently improve over direct-answer prediction and leads to clear drops in some cases (the provided text truncates the full enumeration).

Methodology in Plain English

The framework turns any unlabeled visual data into an automatically checkable puzzle.

1. Split two sources into pieces. Take two images A and B and cut each into K patches (row-major order defines the "correct" order). The same recipe applies to two RGB-D scenes (each contributing K local crops, ordered closest to farthest by depth) and to two videos (each contributing K clips, ordered chronologically).

2. Shuffle everything together. All 2K elements are mixed into one random sequence, so patches from different sources are interleaved.

3. Ask for two ordered lists. The model must output two lists of shuffled indices inside <answer></answer> tags — one per recovered source — and every index must be used exactly once. For images and videos the two groups are interchangeable, so the verifier tries both group-to-source assignments and keeps the better one; 3D uses explicit anchors so no swapping is allowed.

4. Score deterministically. The reward is a format term (0.1 for obeying the answer format) plus a main term combining a grouping score (fraction of elements placed in the correct source) and an ordering score (position accuracy within each matched source). Invalid outputs (repeated, missing, out-of-range, or wrong-sized index lists) get zero task reward; exactly correct answers get 1, and valid imperfect answers get a discounted partial score γ·R̃ so exact recovery is always preferred.

5. Make it harder over time. Training starts with random source pairs (a warm-up for format, grouping, and basic ordering), then switches to hard pairs. CLIP-Hard uses nearest neighbors in CLIP feature space. The main variant, CLIP-Blind, re-ranks CLIP-near candidates with DINO features using score = sim_CLIP − λ·sim_DINO, favoring pairs that are semantically related but differ in fine visual structure. Complexity can also be raised by moving from 2×2 to 2×3 patches per source.

6. Optimize with RL. The authors use a GRPO-inspired group-relative policy objective with LoRA-based RL on Qwen2.5-VL-7B-Instruct, omitting explicit KL penalties and entropy regularization (following the Visual Jigsaw setup). Image tasks are built from the COCO 2014 training split, 3D from ScanNet, video from LLaVA-Video; evaluation uses VLMEvalKit and official protocols.

Why This Matters

Impact on research. GO-SSL shows that a self-supervised RL signal can be made both verifiable and genuinely about cross-instance comparison, a capability that within-instance jigsaw objectives leave unsupervised. Because the reward comes from task construction rather than human annotation or a model judge, the recipe is reproducible and scalable, and the paper's cross-modal instantiations suggest the "group-and-order" principle is not tied to 2D images. The result also reframes a design question for self-supervised post-training: which pretext structure actually transfers to the visual capabilities you care about.

Real-world applications (as motivated by the paper's impact statement):

• Embodied agents and robotics, where accurate spatial and depth-relative judgment is required.
• Assistive systems that depend on fine-grained visual perception of similar-looking objects.
• Visual reasoning tools that must compare and discriminate between instances rather than describe a single image.
• Video understanding pipelines where temporal order and event progression matter.

Industry relevance. The method avoids task-specific annotation, external model judges, and hand-written QA pairs, which lowers the cost of visual post-training. Its moderate budget (1000 steps at batch size 64 for the image setting) and use of standard LoRA-based RL make it practical for teams already post-training open MLLMs, and the released code is available at https://github.com/SEU-VIPGroup/GO-SSL. The paper also cautions that stronger vision-centric MLLMs could be used in safety-critical or surveillance settings outside the scope of this research, and that behavior still depends on the biases of the underlying visual data.

Future Directions

• Efficiency of the group-and-order signal. The conclusion explicitly names improving GO-SSL's efficiency as future work; the paper notes that the O(n²) pairing count is a property of task construction and that a fixed training budget only samples a subset of pairs, so how best to select pairs remains open.

• Scaling to larger MLLMs and specialized visual domains. The authors propose studying how the G&O principle scales beyond the 7B setting and into specialized domains; the current generalization checks cover Qwen2.5-VL-3B and LLaVA-OneVision-7B.

• Better hard-pair and decoy design. CLIP-Blind outperformed CLIP-Hard, and decoy variants improved only the "extra" categories without raising the overall average, leaving room for reward designs that add ambiguity without adding optimization difficulty.

• When explicit reasoning helps. The answer-only versus chain-of-thought comparison found no consistent benefit from CoT and clear drops in some settings, raising the question of which vision-centric tasks (if any) benefit from deliberation during self-supervised post-training.

Target Audience

Researchers and engineers working on multimodal LLM post-training, reinforcement learning with verifiable rewards, and self-supervised visual representation learning. It is most useful for readers who already know what an MLLM and an RL post-training loop are, and who want a concrete, annotation-free recipe for strengthening fine-grained perception and spatial/temporal reasoning. Practitioners deciding between within-instance and cross-instance pretext tasks, or comparing against Spatial-SSRL and Visual Jigsaw baselines, will find the benchmark tables and ablations directly actionable; newcomers to the area can still follow the core idea from the methodology section.

Authors’ abstract

Reinforcement learning (RL) post-training has become an important paradigm for improving multimodal large language models (MLLMs), especially with automatically verifiable rewards. Self-supervised visual pretext tasks provide such rewards without human annotations. However, existing methods mainly exploit within-instance structure, offering limited supervision for cross-instance discrimination and fine-grained comparison. We introduce Group-and-Order Self-Supervised Reinforcement Learning (GO-SSL), a cross-instance jigsaw framework for vision-centric MLLM post-training. Given two visual instances, GO-SSL mixes their local elements and trains the model to group them by instance of origin while recovering the order within each group, coupling instance-level comparison with spatial, temporal, or geometric structure recovery. This paired formulation turns the same unlabeled data into richer verifiable supervision through diverse pairings, shuffles, and a random-to-hard curriculum. We instantiate GO-SSL mainly on cross-image jigsaw tasks and further extend it to 3D depth and video temporal ordering. Across diverse benchmarks, GO-SSL consistently improves over the base MLLM and prior self-supervised post-training baselines, with clear gains in fine-grained perception and spatial understanding. Further evaluations across model scales and families support its generality. These results suggest that cross-instance comparison provides a useful self-supervised RL signal for improving transferable visual understanding in MLLMs. Code is available at https://github.com/SEU-VIPGroup/GO-SSL.