iDETEX: Empowering MLLMs for Intelligent DETailed EXplainable IQA

AI summary

Overview

Research area: Image Quality Assessment (IQA) with Multimodal Large Language Models (MLLMs); specifically the emerging "detailed and explainable IQA" setting defined by the MIPI 2025 Detailed Image Quality Assessment Challenge.

Technical level: Intermediate. The paper assumes familiarity with MLLM fine-tuning (LoRA, ViT unfreezing), object-detection style mAP, and multiple-choice evaluation, but its augmentation ideas are described in largely conceptual terms.

Scope: The paper presents iDETEX, a single fine-tuned MLLM that jointly performs quality grounding, quality perception, and quality description, trained with three task-specific offline augmentation modules, a task-aware data mixing strategy, and an online high-resolution input enhancement.

What This Paper Is About

Traditional IQA methods output a single scalar quality score, which is accurate in some settings but opaque: they do not say where an image is degraded, what low-level attributes are wrong, or why an overall quality judgment was reached. This paper's goal is to train one MLLM that can do all three at once — localize distortion regions, answer fine-grained perceptual questions, and produce structured quality descriptions — despite the fact that annotated explainable-IQA data is small and expensive to label. The authors attack that data bottleneck with augmentation rather than more annotation.

Key Contributions

1. Task-specific augmentation modules for fine-tuning. Three dedicated offline augmentation designs — Spatial Perturbation Augmentation (random cropping plus horizontal flipping with corresponding bounding-box coordinate adjustment) for grounding, Query-Style Aligned Augmentation for perception, and Score-Aware Augmentation (score-driven inference simplification plus granularity-aware label refinement) for description.

2. A Task-Aware Augmented Data Mixing strategy. Augmented samples are inserted back into the original multi-source datasets at controlled ratios (15%, 30%, or 45% for grounding; full replacement of the perception multiple-choice file; modified quality levels for the description files) so heterogeneous supervision can be used in one joint fine-tuning run.

3. iDETEX, a unified MLLM for explainable IQA. Built by replacing the base multimodal model with InternVL3 and fine-tuning it on the augmented mixture, complemented by an online High-Resolution Input Enhancement strategy.

4. State-of-the-art results on ViDA-UGC-Bench and first place in the ICCV MIPI 2025 Detailed Image Quality Assessment Challenge, with ablations isolating each augmentation, the resolution setting, and the choice of foundation model.

Main Findings

• Leaderboard result: Under the team name IH-VQA, the method ranks 1st with a final score of 2.80, versus 2.43 for the second- and third-placed teams (CrazyCat and MediaX), 2.26 for Smart vision group, 1.67 for IVP-Lab, and 0.70 for echoch. The winning row reports Perception Accuracy 0.81, Region mAP 0.40, Distortion mAP 0.12, Description mAP 0.20, Key Distortion Accuracy 0.43, and Image Quality Accuracy 0.83.

• Grounding augmentation is complementary, not individually beneficial: original scores 0.3899 Region mAP / 0.1083 Distortion mAP (avg. 0.2491); horizontal flip alone gives avg. 0.2463; random crop alone gives avg. 0.2351; combining them gives the best avg. 0.2551 (Region mAP 0.3892, Distortion mAP 0.1210). The paper explains this by saying horizontal flipping generalizes distortion appearance while cropping generalizes distortion context.

• 45% horizontal flipping is the best mixing ratio for grounding: 15% gives avg. 0.2494, 30% gives 0.2463, and 45% gives the highest avg. 0.2518.

• Query-style alignment drives the largest perception gain: original training questions score 0.5462 perception accuracy, the self-made (test-format-aligned) version scores 0.7620, shuffle options scores 0.7616, and more options scores 0.7437.

• 10 quality levels is the best description granularity: 5 levels give 0.7857 Image Quality Accuracy, 10 levels give 0.8046, 15 levels give 0.7983, and 20 levels give 0.8004.

• Higher resolution helps, with 2048 max pixel tokens optimal across tasks: grounding avg. rises from 0.2011 (256 tokens) to 0.2203 (512), 0.2304 (1024), and 0.2356 (2048); perception accuracy is 0.7616 (256), dips to 0.7503 (512), then 0.7628 (1024) and 0.7643 (2048); description Image Quality Accuracy is 0.8046 (256), 0.8025 (512), 0.8151 (1024), 0.8172 (2048), and 0.8151 (4096). The authors state the resolution benefit is most pronounced for grounding and less so for the more global perception and description tasks.

• InternVL3-8B-Instruct is the strongest backbone: for grounding with flip plus crop it reaches avg. 0.2768, compared with 0.2551 (Qwen2-VL-7B-Instruct), 0.2707 (Qwen2.5-VL-7B-Instruct), and 0.2570 (InternVL3-9B-Instruct). For perception it reaches 0.7982 versus 0.7620, 0.7709, and 0.7709 respectively. For image quality accuracy it reaches 0.8151 versus 0.8046, 0.7920, and 0.7857.

• Reported training setup: ms-swift framework, LoRA with rank 16, Vision Transformer and cross-modal aligner unfrozen, initial learning rate 4e-5, weight decay 0.01, warm-up ratio 0.03, cosine annealing, single epoch.

Methodology in Plain English

The authors start from a pretrained multimodal model that already understands images and language together, then teach it three related skills at once on the ViDA-UGC dataset. Because each skill stresses something different, they prepare the training data differently for each one.

For grounding (finding distortions), they simulate variation by cropping images to a fraction of their original size and by mirroring them horizontally — and critically, they recompute the distortion bounding boxes so the labels still line up with the changed images. This stops the model from memorizing distortions at fixed positions or sizes.

For perception (multiple-choice questions about visual attributes), they noticed the training questions were phrased differently from the test questions. They rewrote the training questions to match the test format while keeping all the original attribute content, and appended a subset of these to the training file for that task.

For description (producing quality judgments), they made two changes. First, they simplify inference by letting the model reuse scoring supervision from the perception task so it only has to emit the global score at that step instead of inferring everything at once. Second, they refine the rating scale: instead of collapsing the continuous MOS range of 1 to 5 into five labels (bad, poor, fair, good, excellent), they split it into finer internal levels such as 10, 15, or 20, train against those, and then map predictions back to the standard five levels so results stay human-readable.

They then blend all these augmented sources into one jointly fine-tuned training set, with the amount of augmented grounding data swept at 15%, 30%, and 45%. On top of the offline augmentation, they raise input image resolution during fine-tuning, and they swap the base model for InternVL3 because it handles high-resolution inputs and localized regions better than Qwen2-VL. Evaluation follows the ViDA-UGC-Bench protocol: mean Average Precision for grounding, accuracy for perception multiple-choice questions, and three metrics for description.

Why This Matters

The paper argues that IQA should move past black-box scalar scores toward evaluations that are transparent and aligned with how humans actually perceive quality. By showing that a single tuned MLLM can jointly localize distortions, answer fine-grained perceptual questions, and explain quality judgments, it provides evidence that explainable IQA is trainable with modest annotation budgets when augmentation is designed carefully per task.

Real-world applications:

• Image and video compression pipelines that need to know which regions are visibly degraded and why, not just an overall score.
• Camera and imaging ISP tuning, where automatic identification of localized distortion types speeds up quality debugging.
• User-generated content platforms that must triage large volumes of uploads for quality problems.
• Generative image and video systems where the output must be checked for localized artifacts and given an interpretable quality verdict.

Industry relevance: The work comes from Tencent WeChat and was validated in a competitive academic challenge, suggesting direct applicability to large-scale consumer image pipelines where both accuracy and explainability matter for automated moderation and quality control.

Future Directions

• Transfer to low-data regimes. The authors note that training on the large-scale ViDA-UGC dataset raises questions about small-data scenarios, and posit transfer learning — using the pretrained model as a backbone for efficient fine-tuning on smaller, specialized datasets such as niche distortion types — as the primary path to scalability.
• Few-shot learning. The paper explicitly lists exploring complementary few-shot learning techniques as a promising direction.
• Validating transferability. Whether the pretrained model actually transfers well to domain-specific distortions with less annotation remains to be demonstrated.
• Extending the augmentation and evaluation scope. The benchmark covers ten types of UGC distortions; generalization to distortion types and evaluation settings beyond that set is not established in the reported experiments.

Target Audience

Researchers and engineers working on image quality assessment, multimodal large language models, or multi-task vision-language fine-tuning — particularly those interested in data-efficient augmentation strategies and in benchmarks such as ViDA-UGC-Bench. It is also useful for practitioners who need interpretable, region-aware quality signals rather than scalar scores, and for participants or observers of the MIPI 2025 Detailed Image Quality Assessment Challenge.

Authors’ abstract

Image Quality Assessment (IQA) has progressed from scalar quality prediction to more interpretable, human-aligned evaluation paradigms. In this work, we address the emerging challenge of detailed and explainable IQA by proposing iDETEX-a unified multimodal large language model (MLLM) capable of simultaneously performing three key tasks: quality grounding, perception, and description. To facilitate efficient and generalizable training across these heterogeneous subtasks, we design a suite of task-specific offline augmentation modules and a data mixing strategy. These are further complemented by online enhancement strategies to fully exploit multi-sourced supervision. We validate our approach on the large-scale ViDA-UGC benchmark, where iDETEX achieves state-of-the-art performance across all subtasks. Our model ranks first in the ICCV MIPI 2025 Detailed Image Quality Assessment Challenge, demonstrating its effectiveness and robustness in delivering accurate and interpretable quality assessments.