Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images

AI summary

Overview

Research area: multi-modal large language models (MLLMs), structured and abstractive reasoning, and multi-modal relational knowledge (MMRK) images. Technical level: Advanced. Scope: the paper introduces an automatic data engine, the STAR-64K dataset, a two-stage training pipeline with Knowledge-informed GRPO (KGRPO), and evaluation protocols for Structured and Abstractive Reasoning (STAR) over MMRK images.

What This Paper Is About

Current MLLMs struggle to understand and reason over abstractive visual information, especially MMRK images that encode entities, images, text descriptions, and relational edges as node-edge structures. The paper addresses the lack of large-scale high-quality STAR data and the absence of effective training and evaluation frameworks by synthesizing MMRK instruction data and enhancing MLLMs through supervised fine-tuning, preference alignment, and reinforcement learning.

Key Contributions

1. An automatic STAR data engine that synthesizes images with MMRK and builds multi-modal instructions with reliable chain-of-thought thinking for various STAR tasks, using large-scale multi-modal knowledge graphs (MMKGs) to avoid costly manual annotation.
2. A comprehensive two-stage training and evaluation pipeline: Stage 1 supervised fine-tuning for general competency, and Stage 2 targeted optimization using preference alignment (PA) and reinforcement learning (RL), along with dedicated STAR evaluation protocols.
3. Knowledge-informed GRPO (KGRPO), which extends GRPO with a knowledge-informed reward to reward factual knowledge correctness within the chain-of-thought, reducing hallucinations and improving final performance.
4. In-depth experimental exploration across open-source MLLM backbones, including analysis of task transferability, data scalability, design reasonability, and preservation of general capabilities. The abstract reports experiments across 8 open-source MLLMs, and the contribution summary mentions 5 famous open-source MLLM backbones from 3B to 34B.

Main Findings

• Existing mainstream MLLMs fail on STAR tasks. Zero-shot average accuracy is 33.11 for GPT-4V, 40.72 for GPT-4o-mini, and 41.37 for GPT-4o. Qwen2.5-VL-3B zero-shot averages 25.56, and Qwen2.5-VL-7B zero-shot averages 21.24.
• SFT partially unlocks STAR ability. After SFT with single-task data, Qwen2.5-VL-3B reaches 41.76 average accuracy, comparable to or slightly better than Qwen2.5-VL-72B at 38.74 and GPT-4o at 41.37.
• The two-stage pipeline progressively improves performance. For Qwen2.5-VL-3B, S1(Single) averages 41.76, S1(Full) averages 53.24, and S2(KGRPO) averages 63.64. For Qwen2.5-VL-7B, S1(Single) averages 66.06, S1(Full) averages 66.98, and S2(KGRPO) averages 73.06.
• KGRPO outperforms PA methods and vanilla GRPO in Stage 2. The paper reports that on the 7B model, the performance improvement brought by GRPO is 1.3% versus 5.9% for KGRPO, demonstrating the value of the knowledge-informed reward design.
• Single-task training shows limited transferability. Compared to joint training on the full dataset, single-task training generally results in diminished performance across most tasks, with Task #1 being a notable exception.
• Pairwise task combinations reveal mutual enhancement. Tasks 4 and 6 benefit especially from being trained alongside other tasks.
• Data scalability helps most tasks. From 10% to 100% data, most tasks show increasing STAR capability, except Task #1 and Task #4. For Task #1, accuracy fluctuates while CoT quality steadily improves. Task #7 shows a similar pattern: accuracy remains mostly unchanged while CoT quality continues to climb.
• Chain-of-thought prompts matter. Removing CoT prompts from training data consistently degrades STAR performance across all 5 different MLLMs, whether trained on single-task or full multi-task data.
• Entity modality contributes to reasoning. Removing entity images or entity texts from MMRK images noticeably drops performance on Task #1 and Task #7. Omitting entity texts leads to a greater decline, suggesting textual information is more critical.
• General capabilities are retained or improved. On MMMU, two-stage training with STAR-64K preserves and in some domains like arts and business enhances commonsense knowledge. KGRPO shows more pronounced retention and enhancement of commonsense knowledge compared to DPO. Additional commonsense retention experiments on TextVQA and OCRBench are in the appendix.

Methodology in Plain English

The researchers built an automatic data engine that samples subgraphs from three public MMKGs: VisualSem, FB15K-237, and MKG-Y. Each subgraph is limited to a maximum of 9 entities and is processed according to one of 8 seed tasks. The understanding tasks are Entity Counting (EC, Task #1), Relation Counting (RC, Task #2), Image Counting (IC, Task #3), Triple Counting (TC, Task #4), and Subgraph Description (SD, Task #5). The reasoning tasks are Error Detection (ED, Task #6), Entity Reasoning (ER, Task #7), and Relation Reasoning (RR, Task #8).

For each task, the engine modifies the subgraph, visualizes it as an MMRK image using a KG visualization tool such as GraphViz, and generates a question-answer pair where the answer includes a chain-of-thought process and a final answer. The engine produces 8000 data instances per task, split into train/valid/test sets with an 8:1:1 ratio. This yields 51200 training cases, 6400 validation instances, and 6400 test instances, collectively named STAR-64K.

Training proceeds in two stages. Stage 1 performs supervised fine-tuning on the synthesized instruction data. Stage 2 applies targeted optimization on failure cases using either preference alignment methods such as DPO, ORPO, and SimPO, or reinforcement learning methods such as GRPO. KGRPO adds a knowledge-informed reward, r_know, computed as the average over entities in the subgraph that appear in the MLLM output. The final reward is a weighted sum: r = omega * r_acc + (1 - omega) * r_know, where r_acc is the 0-1 accuracy reward for the final answer.

Evaluation uses accuracy for counting tasks and detection/reasoning tasks, comparing predictions against ground-truth answers. For Task #5, which is open-ended, subgraph descriptions are assessed with the same approach as chain-of-thought evaluation. CoT quality and Task #5 are evaluated using the LLM-as-a-Judge paradigm with a stronger LLM as evaluator.

Why This Matters

This work opens a new direction for MLLM research by targeting structured and abstractive reasoning over MMRK images, a capability that remains largely under-explored. It shows that targeted training can enable smaller 3B/7B models to significantly outperform GPT-4o in STAR, suggesting that the main bottleneck is relevant training data rather than model scale alone. The release of STAR-64K, the data engine, the two-stage training pipeline, KGRPO, and the evaluation protocol provides a full-stack resource for future work.

Real-world applications suggested by the paper's focus on MLLMs and MMKGs include:

• Knowledge-intensive visual question answering over entity-relation diagrams.
• Multi-modal knowledge graph understanding and completion for encyclopedic and commonsense knowledge.
• Domain fields where MLLMs are increasingly adopted, such as automatic driving, health care, and agriculture.
• Reducing hallucinations in MLLM reasoning by grounding chain-of-thought in reliable multi-modal knowledge.

Industry relevance: the work was supported by Ant Group and conducted within the Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph. The code is available at https://github.com/zjukg/STAR. The combination of automatic data synthesis, preference alignment, RL-based post-training with knowledge-informed rewards,

Authors’ abstract

Understanding and reasoning with abstractive information from the visual modality presents significant challenges for current multi-modal large language models (MLLMs). Among the various forms of abstractive information, Multi-Modal Relational Knowledge (MMRK), which represents abstract relational structures between multi-modal entities using node-edge formats, remains largely under-explored. In particular, STructured and Abstractive Reasoning (STAR) on such data has received little attention from the research community. To bridge the dual gaps in large-scale high-quality data and capability enhancement methodologies, this paper makes the following key contributions: (i). An automatic STAR data engine capable of synthesizing images with MMRK to build multi-modal instruction data with reliable chain-of-thought thinking for various STAR tasks and (ii). A comprehsive two-stage capability enhancement training framework, accompanied by a suite of evaluation protocols tailored to different STAR tasks. Based upon these contributions, we introduce STAR-64K, a dataset comprising 64K high-quality multi-modal instruction samples, and conduct experiments across 5 open-source MLLMs. Experimental results show that our two-stage enhancement framework enables smaller 3B/7B models to significantly outperform GPT-4o in STAR. Additionally, we provide in-depth analysis regarding the effectiveness of various designs, data transferability, and scalability.