LISTEN to Your Preferences: An LLM Framework for Multi-Objective Selection

AI summary

Overview

• Research area: Natural Language Processing applied to decision support and multi-objective optimization, specifically using LLMs as agents for preference-driven selection.
• Technical level: Intermediate. The framing is accessible (natural-language preferences, iterative LLM prompting), but the paper includes formal notation for the selection problem, a linear utility model, and two algorithms presented as pseudocode.
• Scope: The paper introduces LISTEN, an agentic LLM framework with two algorithms (LISTEN-U and LISTEN-T) for selecting a single best item from a large set of multi-attribute candidates, and evaluates it on three decision domains against four baselines.

What This Paper Is About

A person facing a large set of options described by multiple competing attributes (flight itineraries, headphones, exam schedules) usually cannot state a precise utility function, and manually inspecting every option is slow and prone to error. The paper's goal is to let that person describe their priorities in ordinary natural language, then have an LLM act as an iterative decision-making agent that narrows the candidate set down to one item, while respecting the LLM's context-window limits and a limited budget of calls. The authors frame this as the "post-solution" problem: choosing among the many trade-off solutions already produced by a multi-objective optimizer.

Key Contributions

1. The LISTEN framework. An agentic, iterative decision loop in which an LLM observes a subset of candidates, reasons about trade-offs from a user's natural-language utterance, updates an internal preference representation, and takes an action (proposing a utility function or selecting a batch winner).
2. Two algorithms within the framework. LISTEN-U, a parametric method that has the LLM iteratively refine a weight vector for a linear utility function over normalized numerical attributes, and LISTEN-T, a non-parametric tournament method that has the LLM select a champion from each of several random batches and then pick an overall winner from those champions.
3. A novel concordance metric. A dataset-level diagnostic that measures how often the optimal item under randomly sampled linear utility functions falls inside the human-ranked set, used to predict when a parametric approach like LISTEN-U will succeed or fail.
4. Human preference benchmarks across three domains. Datasets with a natural-language preference utterance paired with expert-ranked ground-truth items for flight booking, headphone shopping, and university final exam scheduling, released alongside code.

Main Findings

• LISTEN methods match or beat non-human baselines. Across the four datasets and two LLMs, the LISTEN algorithms tend to outperform uniform random selection, the normalized average score baseline, and single-call full-batch LLM ranking.
• LISTEN-U excels when preferences align with a linear utility. The method performs significantly better than alternatives on datasets with high concordance, and the paper reports that LISTEN-U may underperform when concordance is low unless the prompt or utility form is adjusted.
• LISTEN-T is the more robust option overall. Because LISTEN-T makes no parametric utility assumption, it delivers consistent performance across the range of problems tested.
• Concordance varies widely across the datasets. Exam Scheduling scored 0.0015 ± 0.0008, Flights CHI→NYC scored 0.0025 ± 0.0011, Headphones scored 0.0552 ± 0.0047, and Flights Ithaca→Reston scored 0.3357 ± 0.0095 (reported as mean ± 2SE, approximating a 95% CI).
• Removing hard constraints raises concordance and helps LISTEN-U. On a "Headphones-Soft" prompt variant that drops hard constraints such as a strict budget and required categorical features, concordance rose from 0.055 to 0.244, and LISTEN-U's performance increased significantly, while LISTEN-T stayed robust across both prompts.
• Beating full-batch LLM ranking. LISTEN's smaller per-iteration ranking tasks reduce task complexity, multiple iterations give the LLM repeated chances to focus on trade-offs among high-quality options, and for some dataset/LLM combinations full-batch ranking would not fit in the context window at all.
• Humans re-rankers were highly variable. The human-rerank baseline showed large variance in normalized average rank, partly because human top choices were more often unranked items (which receive a normalized average rank of approximately 0.5) and partly because only 4 samples were collected. LISTEN methods more consistently selected items that were in the ranked set, though the authors note LISTEN may be less likely to optimize very fine trade-offs the decision-maker prefers.
• Performance was robust to batch size. The authors report a surprisingly high level of consistency in performance across batch sizes for each LLM/dataset pair, and state that further analysis is needed to test this at even larger batch sizes.
• Setup specifics. Experiments used Llama-3.3-70B-Versatile and Gemini 2.5 Flash-Lite as the preference oracle, with all algorithmic runs replicated 40 times using different random seeds; the LISTEN methods used 25 iterations and batch size 32, and every algorithm had n=40 runs except baseline/human-rerank (n=4) and baseline/zscore-avg (n=1).

Methodology in Plain English

The researchers treat the LLM as a preference oracle that is never fine-tuned. Every query follows a structured prompt with five parts: a persona ("You are an expert university registrar"), definitions of the item attributes, the user's natural-language priorities, the candidate solutions to evaluate, and output-format instructions (for example, a JSON object, or a fixed string format for LISTEN-T).

For LISTEN-U, the LLM first proposes an initial weight vector over the numerical attributes from the user's utterance. All numerical attributes of every candidate are normalized to a [0, 1] scale, and the algorithm scores each item by a dot product of weights and normalized attributes, picking the top scorer. In each later iteration, the LLM is shown the current weights plus the full, unnormalized description of the current best item (including categorical and textual attributes, so it can reason about context like "the price is too high for this particular brand"), critiques it, and returns refined weights; the whole set is rescored with the new weights. Weights serve as a compact belief state over the user's latent preferences.

For LISTEN-T, the process is a tournament requiring at least 3 LLM calls. It runs R = T − 1 preliminary rounds, each sampling a batch of B items uniformly at random, with the LLM choosing one batch champion per round. A final playoff then presents all champions to the LLM to pick the overall winner. Requiring T ≥ 3 ensures the playoff compares at least two distinct champions.

Ground truth came from a domain expert who first wrote their decision criteria as a natural-language directive (the utterance U) and then manually ranked a subset of the most relevant items. Because ranking all N items is impractical, the experts ranked only a small subset. Flights were generated with the fast_flights Python package for two scenarios (Chicago to New York, and Ithaca to Reston); headphones data was scraped from Amazon.com using the Scrapingdog API; and the exam scheduling set consisted of 4,938 Pareto-optimal schedules generated with the ParEGO multi-objective algorithm, based on mixed-integer programming methods of Ye et al. (2026).

Evaluation used normalized average rank: if an algorithm's selected item is in the human top-m list, its rank is its position; otherwise it receives a shared average rank of (m + 1 + N)/2, and ranks are divided by N so the metric falls in (0, 1] with lower being better. Concordance was computed by sampling 10,000 random linear utility functions with weights drawn independently from U[−1, 1] and measuring the fraction for which the optimal item lies within the m-item human-ranked set.

Baselines were uniform random selection; normalized average score (z-scored numerical attributes with minimization attributes negated, then equal-weighted average, ignoring categorical and textual information); full-batch LLM ranking of the entire set in one call with randomized seed and item order; and human re-rankers, distinct from the ground-truth decision-maker, who ranked the top 20 items they believed best matched the utterance using a filter-and-sort system.

Why This Matters

Impact on research. The paper targets a gap the authors identify between work on the "pre-solving" phase—formulating optimization problems from natural language and configuring solvers—and the "post-solution" problem of helping a user navigate a potentially huge set of Pareto-optimal trade-off solutions. It also extends existing findings about LLM preference instability (sensitivity to prompt framing and divergence from human behavior) into new multi-objective settings, and offers a measurable dataset property (concordance) that predicts when a parametric LLM-guided approach will work.

Real-world applications (drawn from the domains the paper addresses):

• Selecting a flight itinerary from a commercial flight search engine, where itineraries vary by price, duration, layovers, airline, departure time, and ground travel distance to the final destination.
• Choosing a product on an e-commerce site, such as headphones with mixed numerical, categorical, and textual attributes.
• Scheduling university final exams from thousands of Pareto-optimal schedules, trading off direct conflicts, exam-density patterns, back-to-back exams, and average time of the last exam.
• Multi-objective engineering and science design problems, which the cited surveys argue make up most real-world engineering design problems.

Industry relevance. The framework requires no fine-tuning and no external tools, relying only on structured prompting, which makes it cheap to prototype against any frontier or open-weight LLM; the paper explicitly tests two different models. For travel, retail, and operations-research software vendors, LISTEN suggests a way to add a natural-language "tell me what you care about" layer on top of an existing optimizer's solution set rather than replacing the optimizer. The cost constraint the algorithms are built around—limited LLM calls and finite context windows—is directly relevant to production economics. The concordance metric also gives practitioners a way to pre-screen whether a parametric utility approach is likely to work on their problem before deploying it.

Future Directions

• Better understanding of preference variability. The ground-truth rankings reflect a single expert per domain and may not generalize; the authors observed high variability among human re-rankers relative to the ground truth, and suggest gathering more data to understand the range of preferences that correspond to one utterance, potentially returning sets of items rather than a single one.
• Broader domain testing. LISTEN was evaluated on three domains; the authors explicitly name apartment hunting, healthcare scheduling, and logistics planning as settings whose performance is yet to be tested at scale.
• Richer utility representations. LISTEN-U's reliance on a linear utility function is named as a key limitation for highly non-linear preferences; the authors propose exploring non-linear or piecewise functions and hybrid methods combining LLM guidance with structured optimization.
• Larger batch-size analysis. Despite the observed consistency across batch sizes 2–32, the authors state there would be value in further analysis to see how robust this finding is at even larger batch sizes.

Target Audience

Researchers and graduate students in NLP, LLM agents, and AI for decision-making; operations-research and preference-learning researchers interested in the post-solution selection problem; and practitioners building decision-support or recommendation features on top of multi-objective optimizers in travel, e-commerce, and scheduling. Readers with no background in optimization can follow the high-level framing, but the algorithm details and the concordance metric assume some comfort with utility functions and multi-objective terminology.

Authors’ abstract

Human experts often struggle to select the best option from a large set of items with multiple competing objectives, a process bottlenecked by the difficulty of formalizing complex, implicit preferences. To address this, we introduce LISTEN (LLM-based Iterative Selection with Trade-off Evaluation from Natural-language), an agentic LLM-based framework that treats the LLM as a decision-making agent capable of iteratively refining its internal preference model and taking actions (e.g., proposing utilities or selecting candidates) to maximize alignment with a user's implicit goals. To operate within LLM constraints like context windows and inference costs, we propose two iterative algorithms: LISTEN-U, which uses the LLM to refine a parametric utility function, and LISTEN-T, a non-parametric method that performs tournament-style selections over small batches of solutions. Evaluated on diverse tasks including flight booking, shopping, and exam scheduling, our results show LISTEN-U excels when preferences are parametrically aligned (a property we measure with a novel concordance metric), while LISTEN-T offers more robust performance overall. This work explores a promising direction for steering complex multi-objective decisions directly with natural language, reducing the cognitive burden of traditional preference elicitation. Code is available at https://github.com/AdamJovine/LISTEN; data is available at https://huggingface.co/datasets/AdamJovine/LISTEN-benchmark.