Agile Deliberation: Concept Deliberation for Subjective Visual Classification

AI summary

Overview

Research area: Human-in-the-loop computer vision / human-AI interaction, specifically interactive systems for training image classifiers on subjective visual concepts.

Technical level: Intermediate. Readers should be comfortable with vision-language models (VLMs), prompt optimization, and human-in-the-loop ML pipelines, but the paper's framing is largely conceptual and design-oriented.

Scope: The paper presents Agile Deliberation, a two-stage interactive framework (concept scoping and concept iteration) that helps users articulate, refine, and operationalize subjective visual concepts into VLM-prompted image classifiers, evaluated through 18 live user sessions.

What This Paper Is About

Most visual classification systems assume that the concept to be recognized (e.g., "dog," "car") is objective and well-defined before labeling begins. But many real-world applications—content moderation, content curation—involve subjective concepts like "healthy food" or "unsafe imagery" where users start with only a vague intuition and must refine it through trial and error. This paper formalizes that refinement process, which the authors call concept deliberation, and builds an interactive system that guides users through it while simultaneously training an image classifier aligned with their evolving intent.

Key Contributions

1. Empirical characterization of concept deliberation. Through structured interviews with five content moderation experts and qualitative coding of twenty expert-authored concept definitions, the authors identify common strategies experts use to clarify subjective concepts—scoping the concept first, then iteratively refining it through borderline cases.

2. The Agile Deliberation framework. A two-stage human-in-the-loop system: (1) concept scoping, which decomposes a vague concept into a structured hierarchy of positive and negative subconcepts, and (2) concept iteration, which surfaces semantically borderline images for user labeling and automatically refines the concept definition as a VLM prompt.

3. A semantically grounded borderline retrieval method. Rather than classical uncertainty sampling (which relies on calibrated model probabilities), the system identifies ambiguity in semantic space—generating borderline queries, retrieving candidate images, clustering them via dictionary learning, selecting clusters with a multi-armed bandit, and mining subsets that probe a single coherent ambiguity dimension.

4. A live-session evaluation protocol. Because subjective concepts have no static ground truth, the authors depart from offline benchmarks and evaluate via 18 sessions (9 participants × 2 systems × ~90 minutes), measuring classifier F1 against each participant's own held-out labels plus survey and interview data.

Main Findings

• Agile Deliberation beats automated baselines. Participants using the framework achieved F1 scores roughly 10.5% higher than zero-shot VLM classifiers and about 7.5% higher than Modeling Collaborator (an automated LLM-based concept decomposition baseline), driven mainly by precision gains.

• It also beats manual deliberation. Relative to each system's own zero-shot baseline, Agile Deliberation improved F1 by 11% on "paid to play" and 10% on "healthy food," whereas manual deliberation yielded 8% and −3% respectively. Afternoon gains were larger when participants held nuanced interpretations that deviated from generic VLM priors.

• Users explored more diverse ambiguities. Manual users averaged 7.3 search queries and often fixated on one type of ambiguity; Agile users surfaced a wider range of edge cases (e.g., non-food subjects, high-carb meals, food still being prepared) without having to invent queries themselves.

• Lower cognitive burden. Participants reported significantly less effort to achieve good performance (M = 3.11 vs. 4.67, p < .05) and fewer negative emotions like stress or irritation (M = 1.67 vs. 3.00, p < .05). All nine participants preferred Agile Deliberation.

• Performance trends upward across iteration rounds. F1 fluctuated but generally rose across rounds of concept iteration, though session length capped the number of rounds, limiting how far the trend could be observed.

• Automated refinement alone is insufficient. Modeling Collaborator, which enriches definitions without user feedback, produced only modest gains—indicating that iterative human deliberation, not just LLM decomposition, drives the improvement.

Methodology in Plain English

The researchers first talked to real content moderators to understand how experts actually pin down fuzzy concepts, then turned those strategies into software.

Stage one — scoping. The user types a concept name (e.g., "healthy food"). The system uses an LLM to break it into up to three "unit concepts" (e.g., food, nutritional quality), and for each unit proposes candidate positive and negative subconcepts (e.g., fresh fruit vs. fried fast food). The system retrieves representative images for each candidate so the user can decide what to keep, discard, or flip. The output is a structured textual definition.

Stage two — iteration. That definition becomes a prompt for a VLM classifier. Each round, the system generates borderline search queries (e.g., "salad with heavy mayo dressing"), retrieves images, removes duplicates, and clusters the rest by visual feature using dictionary learning. A multi-armed bandit (UCB rule) picks which cluster to show next based on how informative previous clusters proved. Within that cluster, the system asks a VLM to summarize each image in one sentence, embeds the summaries, and selects a small set that forms a tight cluster in embedding space—so each batch probes one clean ambiguity dimension.

The user labels those images and can add short comments. An automatic prompt-optimization module turns the comments into full rationales, generates several candidate revised definitions, scores each against all user labels collected so far by F1, and greedily picks the best one. Greedy selection was chosen deliberately over beam search or MCMC so users can inspect how their definition changed.

Evaluation. Since there is no fixed ground truth, nine participants each did two 90-minute sessions—one with Agile Deliberation, one with manual deliberation—on two concepts ("paid to play" and "healthy food"). Each built a definition, then annotated 200 held-out images so the researchers could measure agreement between participant and classifier. Surveys and semi-structured interviews captured subjective experience. The system runs on Gemini-Pro 2.5 (decomposition) and Gemini-Flash 2.5 (everything else), with no fine-tuning.

Why This Matters

The paper reframes a widely assumed precondition of supervised learning—that the concept is known before labeling begins—as an interactive, iterative problem to be solved jointly by user and system. It matters because the fastest-growing uses of vision classifiers (moderation, curation, personalization) are exactly the ones where no objective ground truth exists.

Real-world applications:

• Content moderation. Moderators can encode nuanced, jurisdiction-specific or policy-specific interpretations of "harmful" or "clickbait" content into classifiers, rather than relying on generic models that miss edge cases.
• Content curation and recommendation. Individual users or editors can define subjective qualities like "healthy," "aesthetic," or "on-brand" and get classifiers that reflect their own taste.
• Trust and safety policy enforcement. Teams can proactively design classifiers that anticipate emerging harms rather than react to them after they appear.
• Domain-expert knowledge capture. Radiologists, art historians, or quality inspectors whose judgments resist simple rules can externalize tacit criteria into reusable definitions.

Industry relevance: The framework relies only on publicly available foundation models and standard retrieval infrastructure, avoiding fine-tuning so that domain experts without compute resources can use it. The reduction in cognitive load and the unanimous user preference suggest it is deployable for non-expert operators—a meaningful step for organizations that need to scale subjective judgment across many reviewers without losing consistency.

Future Directions

• More user studies and longer deliberation. The evaluation involved only nine participants and two concepts; the authors explicitly note the cost of live sessions limited coverage, and the upward F1 trend suggests more iterations could yield further gains.
• Same-concept within-subject comparison. The current design assigns different concepts to different systems to avoid bias from a participant having already internalized a concept, which prevents clean head-to-head F1 comparisons. A design that allows paired comparison remains open.
• Richer search than greedy prompt selection. Greedy selection was chosen for transparency and latency, but more expressive optimizers (beam search, tree search) could improve final definitions if latency and inspectability can be preserved.
• Generalization beyond vision, and beyond two domains. The pipeline is largely prompt-based and could plausibly extend to audio, video, or text classification, and to concepts in medicine, law, or design where subjective judgment is even higher-stakes.

Target Audience

Researchers and practitioners working at the intersection of human-computer interaction and machine learning, especially those building interactive labeling or model-training tools. It is also valuable for product teams in trust-and-safety, content moderation, and curation who need classifiers for concepts that resist objective definition, and for HCI researchers interested in how users articulate and refine their own mental models through AI-assisted workflows.

Authors’ abstract

From content moderation to content curation, applications requiring vision classifiers for visual concepts are rapidly expanding. Existing human-in-the-loop approaches typically assume users begin with a clear, stable concept understanding to be able to provide high-quality supervision. In reality, users often start with a vague idea and must iteratively refine it through "concept deliberation", a practice we uncovered through structured interviews with content moderation experts. We operationalize the common strategies in deliberation used by real content moderators into a human-in-the-loop framework called "Agile Deliberation" that explicitly supports evolving and subjective concepts. The system supports users in defining the concept for themselves by exposing them to borderline cases. The system does this with two deliberation stages: (1) concept scoping, which decomposes the initial concept into a structured hierarchy of sub-concepts, and (2) concept iteration, which surfaces semantically borderline examples for user reflection and feedback to iteratively align an image classifier with the user's evolving intent. Since concept deliberation is inherently subjective and interactive, we painstakingly evaluate the framework through 18 user sessions, each 1.5h long, rather than standard benchmarking datasets. We find that Agile Deliberation achieves 7.5% higher F1 scores than automated decomposition baselines and more than 3% higher than manual deliberation, while participants reported clearer conceptual understanding and lower cognitive effort.