First Things First: Teaching LLM-Based Agents to Prioritize Must-Haves before Nice-to-Haves

AI summary

Overview

Research area: Multimodal large language model (MLLM) agents, requirement-aware reasoning, reinforcement learning from verifiable rewards, and benchmark construction. The paper is listed under the arXiv category Computer Vision (arXiv:2609.05224v2 [cs.CV]).

Technical level: Intermediate. Readers should be comfortable with multimodal LLM evaluation, supervised fine-tuning, and reinforcement learning concepts such as GRPO, reward shaping, and KL regularization.

One-sentence scope: The paper builds a 3,649-instance benchmark (FTF-bench) showing that current MLLMs fail when they must separate must-have from nice-to-have user requirements, and proposes a multi-objective reinforcement learning method (FTF-RL) that substantially improves this capability.

Affiliation note from the paper: the authors are at the School of Computer Science, Shanghai Jiao Tong University, with ByteDance also listed; the work was supported by the National Natural Science Foundation of China (62406188).

What This Paper Is About

Real user requests in service settings ("book a non-smoking hotel for two people, preferably with breakfast") mix hard requirements with soft preferences, and a correct agent must satisfy the hard ones first. The paper shows that state-of-the-art MLLMs largely cannot do this: they misinterpret requirements, violate must-have conditions, and return invalid answers. The goal is to measure this failure systematically and to train models to reason about requirement priority.

Key Contributions

1. FTF-bench, a benchmark of 3,649 image–requirement pairs drawn from e-commerce, booking, and maps/ride-hailing interfaces, where each task is organized around a necessity hierarchy of must-have and nice-to-have requirements.
2. A three-scenario task formulation: single-answer (exactly one candidate satisfies all must-haves), multiple-answer (two or more candidates satisfy must-haves and must be ranked by prioritized nice-to-haves), and unanswerable (no candidate satisfies the must-haves, so the agent should refuse or abstain).
3. FTF-RL, a reinforcement learning method with a rule-based multi-objective reward that scores format compliance, final-answer correctness, and correct classification of requirements into must_have and nice_to_have (measured by Macro-averaged F1), while encouraging intermediate reasoning.
4. Evidence of transfer: models trained only on FTF-bench improve on unrelated logical and mathematical reasoning benchmarks (LogicVista, MathVision, InfoQA), plus an ablation isolating the contribution of each reward component.

Main Findings

• Large gap between direct and guided performance: The paper compares a "Direct" setting (models read the original colloquial request) with an "Upper" setting (models are given gold requirement labels separating must-haves from nice-to-haves). Upper exceeds Direct in most scenarios, which the authors read as evidence that the dominant error source is requirement parsing and prioritization rather than visual perception.
• Proprietary models lead but still fail: Gemini-2.5-pro reaches an average of 84.26 (Upper) and 81.75 (Direct); GPT-5 reaches 82.49 (Upper) and 80.86 (Direct); GPT-o3 reaches 82.33 (Upper) and 79.68 (Direct); Doubao-1.6-seed reaches 82.24 (Upper) and 80.59 (Direct); Claude-Sonnet-4.5 reaches 78.01 (Upper) and 77.51 (Direct).
• Open-source models degrade sharply without guidance: Qwen2.5-VL-7B-Instruct averages 57.69 (Upper) versus 21.05 (Direct); Qwen2.5-VL-72B-Instruct averages 77.60 (Upper) versus 34.48 (Direct); Qwen2.5-VL-32B-Instruct averages 67.72 (Upper) versus 66.57 (Direct). LLaMA-4 averages 55.03 (Upper) and 51.99 (Direct), while LLaVA variants are lower still (LLaVA-OneVision-7B at 43.38 Upper / 19.73 Direct).
• Scaling is non-monotonic: Within Qwen2.5-VL, the 7B and 72B models fail badly in direct reasoning while the 32B variant does better. The paper attributes this to two distinct failure modes: the 7B model lacks capacity to distinguish must-haves from nice-to-haves, while the 72B model overfits to the surface form of the prompt and promotes optional requirements to mandatory ones.
• Unanswerable tasks trigger over-accommodation: In the unanswerable setting, where every option should be rejected, most MLLMs still try to produce an answer instead of abstaining.
• FTF-RL yields large in-domain gains: On the held-out 10% subset, Qwen2.5-VL-7B-Instruct improves from an average of 39.78 to 55.80 (+16.02), including a jump from 32.52 to 58.90 (+26.38) in the multiple-answer scenario and 46.38 to 57.97 (+11.59) in single-answer, with a decrease of 1.64 in the unanswerable scenario (44.26 to 42.62). Qwen2.5-VL-3B-Instruct improves from 38.95 to 45.85 (+6.90); LLaVA-OneVision-7B from 39.86 to 46.32 (+6.46); LLaVA-1.5-13B from 42.93 to 50.00 (+7.07).
• Gains transfer to other reasoning benchmarks: Qwen2.5-VL-7B-Instruct moves from 43.40 to 47.43 (+4.03) on LogicVista, 24.67 to 25.65 (+0.98) on MathVision, and 65.35 to 67.81 (+2.46) on InfoQA. Qwen2.5-VL-3B-Instruct gains 3.58, 1.31, and 2.31 respectively; LLaVA-1.5-13B gains 6.38, 2.62, and 0.35.
• Every reward component matters: Removing the requirement reward drops FTF-bench validation from 55.8 to 52.4, LogicVista from 47.4 to 44.7, MathVision from 25.7 to 24.7, and InfoQA from 67.8 to 60.3. Removing the accuracy reward drops FTF-bench validation to 48.3 (a decline of 7.5) and InfoQA to 61.2. Removing the format reward drops FTF-bench validation to 54.5.
• Recognized failure types: Table 1 illustrates a must-have conflict with no refusal, and a misclassification of a nice-to-have requirement as must-have. In the second example, the model refuses, while the expected behavior is to treat the rating criterion as optional and list all rooms satisfying price and capacity requirements, possibly ranked by rating.

Methodology in Plain English

The authors first collected screenshots of real consumer applications, sampling different interface states such as homepages, product pages, and shopping carts. A large-model-driven pipeline then generated requirements and ground-truth answers for each image, matching one of the three task types. Structured requirements were rewritten by a model into colloquial, single-utterance user queries that still preserve the must-have/nice-to-have hierarchy. Four trained annotators independently verified every image–query pair for requirement correctness, answer consistency, and the separation between mandatory and optional constraints, resolving disagreements by discussion.

Formally, each sample is a tuple of an image, a set of candidate objects, a set of must-have requirements, and a set of nice-to-have requirements. A candidate satisfies all must-haves only if every must-have predicate evaluates to true; the set of such candidates determines the scenario (size one, size two or more, or empty). For multi-answer cases, nice-to-have requirements carry an explicit priority order, and candidate sets are filtered iteratively from highest to lowest priority.

For training, the framework samples G candidate outputs from an old policy, scores each with a multi-objective rule-based reward, and optimizes the policy to maximize advantage while a KL divergence term keeps the updated policy close to a reference policy. The three reward components are a format reward (correct XML-style tags for requirement classification, thinking, and answer), an accuracy reward (an MLLM judge checks semantic equivalence between the extracted answer and the gold answer, since tasks are fill-in-the-blank), and a requirement reward (scaled Macro-F1 over the must_have and nice_to_have classes). Training uses GRPO with a KL-penalty coefficient of 10⁻², a global batch size of 128, a learning rate of 10⁻⁶, AdamW optimization, full-shard FSDP, rollout temperature 1.0, and n = 5 sampled candidate responses. Ninety percent of FTF-bench is used for training and ten percent held out for evaluation; the paper states that the 10% subset was verified to be representative of the full benchmark in Appendix H.

Why This Matters

Impact on research. The paper reframes a common assumption in agent evaluation: that users give unambiguous instructions and agents only need to execute them. It argues that the bottleneck for realistic agents is not perception or computation but requirement comprehension, and it provides both a diagnostic benchmark with an "Upper" reference condition and a training recipe that narrows the gap to that upper bound. The demonstrated transfer to LogicVista, MathVision, and InfoQA suggests requirement-aware supervision may be a general-purpose lever for improving multimodal reasoning, not just a domain-specific fix.

Real-world applications

• E-commerce assistants that must filter products by hard constraints before ranking by preferences.
• Hotel, flight, and restaurant booking agents that must detect when no listing meets mandatory criteria and say so.
• Maps and ride-hailing agents that must respect hard constraints such as capacity or price limits while optimizing soft preferences.
• Customer-service chatbots and GUI-navigating agents that need to refuse infeasible requests instead of hallucinating a plausible option.

Industry relevance. The failure modes documented here (over-constraining by treating preferences as rules, under-constraining by ignoring mandatory conditions, and failing to abstain) map directly onto deployed assistant products where a fabricated recommendation is worse than a refusal. The paper reports that training on 90% of FTF-bench improved multiple-answer accuracy for Qwen2.5-VL-7B-Instruct by more than 26 points, indicating that this capability is learnable rather than an inherent limitation, and that abstention behavior can be shaped.

Future Directions

• Extending FTF-bench to more domains and to settings where user requirements evolve over time, as the limitations section explicitly proposes.
• Scaling the training data beyond the current size of FTF-bench to characterize the upper limit of the requirement-aware supervision effect and how far it transfers.
• Investigating the non-monotonic scaling behavior, in which a 72B model overfits to prompt surface form while a 32B model handles priority better, to determine whether instruction-tuning or reward design is the cause.
• Refining the reward design, given that removing the requirement, format, or accuracy reward each produced consistent declines across FTF-bench and the three transfer benchmarks, and that the unanswerable scenario regressed for one trained model.

Target Audience

Researchers and engineers working on multimodal agents, instruction following, and reinforcement learning for LLMs, particularly those building or evaluating assistants for e-commerce, travel booking, and maps or ride-hailing. The paper is also useful for benchmark designers interested in necessity hierarchies and abstention, and for practitioners who need concrete evidence about which model scales and training signals improve requirement-aware behavior. Code and dataset are stated to be available at https://github.com/claire62/FTF-RL.

Authors’ abstract

Recent progress in multimodal large language models (MLLMs) has fueled significant enthusiasm in their potential to act as autonomous agents for real-world tasks. However, scenarios requiring agents to fulfill users' complex, structured requirements remain largely underexplored. In this work, we examine reasoning tasks under three distinct requirement scenarios: (i) Must-have requirements uniquely determine a unique feasible solution; (ii) Multiple answers satisfy the must-have requirements and are prioritized via the nice-to-have requirements; and (iii) No candidate solution satisfies the must-have requirements, in which case the agent should abstain from generating a response. We evaluate state-of-the-art MLLMs on 3,649 carefully constructed problems that reflect realistic service scenarios, including e-commerce, booking, and map-based or ride-hailing. Our evaluation reveals that existing MLLMs exhibit catastrophic failures in all scenarios. They frequently misinterpret task requirements, violate must-have requirements, and produce invalid solutions. To address this critical gap, we propose First Things First Reinforcement Learning FTF-rl that explicitly optimizes reasoning over multi-priority user requirements. Experimental results show that our method substantially improves the task success rate compared to strong baselines. Moreover, FTF-rl yields general effectiveness on popular logical and mathematical reasoning tasks, including LogicVista, MathVision, and InfoQA. Our findings suggest that enhancing requirement-aware reasoning capability provides a simple yet effective pathway to improve generalization of MLLM agents. Code and dataset are available at https://github.com/claire62/FTF-RL.