Interpretable Feature Discovery from Unstructured Data Using LLMs and Statistical Choice Models

AI summary

Overview

Research area: Machine learning / statistics — interpretable feature discovery from unstructured text using large language models (LLMs) combined with statistical choice models (stat.ML).

Technical level: Intermediate. The paper combines an intuitive pipeline (pairwise LLM comparisons) with a non-trivial latent-variable choice model and consistency proofs, but the core ideas are explained without heavy jargon.

Scope (one sentence): The paper proposes and evaluates an end-to-end method that has an LLM compare items in pairs, articulate the distinguishing features in words, and then fits a statistical choice model to those written rationales in order to recover interpretable, dataset-specific features with per-item scores.

What This Paper Is About

Extracting features from unstructured data such as essays, résumés or forum arguments usually means using opaque embeddings, which are unsuitable when decisions must be transparent. Hand-building interpretable rubrics is expensive and often infeasible, and simply asking an LLM to read a whole dataset and list the relevant features runs into context limits and diluted attention. The paper's goal is to automatically discover interpretable, decision-relevant features — plus a score for every item on every feature — while correcting for the LLM judge's biases and flagging which discovered features can be trusted.

Key Contributions

1. A pairwise "rationale-as-data" pipeline. Instead of one long prompt over the dataset, an LLM compares items along a comparison graph, naming the features on which each pair differs and which item is stronger on each; a second LLM call groups these free-text descriptions into K shared features, producing a sparse verdict tensor with entries in {−1, 0, +1}.

2. A statistical choice model for aggregating rationales. Inspired by Bradley–Terry models, it treats each verdict as a two-stage outcome — whether a feature is cited at all (driven by the score gap through an even function f = log(2 cosh) plus a feature-specific baseline citation propensity γ_k) and, conditional on citation, which item wins (driven by the score gap plus a shared positional-bias term β). Fitting it by penalized maximum likelihood yields per-item, per-feature scores, confidence intervals, and diagnostics.

3. Theory for a growing-parameter likelihood. The authors prove (Theorem 1, for K = 1, under assumptions A1–A3 and A6) that in a dense regime where every pair is compared at least once and at most a fixed number of times, the non-convex negative log-likelihood admits a local maximizer that is consistent for every item score at once, with ‖ŝ − s*‖∞ ≤ C√(log n / n) and the same rate for γ and β; it is the unique stationary point in a neighborhood. Theorem 2 in Appendix B generalizes this to comparison-level covariates and coefficient vectors.

4. Two reliability diagnostics and an auditing extension. A condition number κ_k of the score block of the Fisher information and a reliability coefficient ρ_k (the fraction of observed score spread attributable to true between-item differences rather than estimation noise) filter out features whose scores are poorly pinned down (features with κ_k > κ_max or ρ_k < ρ_min are dropped before a final refit). Appendix A extends the framework to auditing an LLM's own pairwise decisions, giving a holistic score that decomposes over interpretable features.

Main Findings

• Feature recovery on synthetic hiring data: On 200 cover letters built from 18 engineered ground-truth "dials" (16 relevant recovery targets plus two controls: skill coverage, a redundant composite, and prose fluency, irrelevant to the job), the method tracks all 16 ground-truth features versus 13 for Direct LLM, and achieves 100.0% semantic recovery versus 81.2%.

• Attribution and leakage: 16 of 21 discovered features are attributable to a single ground-truth feature, versus 13 of 21 for Direct LLM, and leakage is lower (0.090 versus 0.109). Direct LLM is cheaper, however ($0.26 versus $3.45).

• Gains are not just from multi-agent scale: At roughly matched cost, Direct LLM–Multi-score ($3.42) still tracks the same 13 dials, so the three missing dials are a discovery failure, not a scoring failure. Local LLMs and Local LLMs–Multi-score reach 75.0% tracked, and the paper reports statistically significant gains in tracking, attribution and leakage over Local LLMs.

• Statistical scoring and filtering matter: The ablations keep the same 21 discovered features but use LLM scoring instead of the statistical model; attribution falls to 13–14 of 32 features with leakage of 0.103–0.104, despite slightly higher Stat.R. (0.860–0.868 versus 0.850) and roughly twice the cost. Filtering alone raises the attributable fraction under multi-agent scoring from 14 of 32 to 14 of 21 while preserving complete tracking and semantic recovery; statistical scoring on the same 21 features then raises it to 16.

• Real-world debate application: On Reddit's ChangeMyView, running the pipeline on 300 training exchanges produces 29 reply-side features, of which the diagnostics retain 16. The features are human-recognizable (e.g. concrete illustrations, structured rebuttals, conceptual clarification, confrontational tone) and need no hand-engineering.

• Which argumentation techniques persuade: Features related to evidence and cases show stronger positive associations with persuasion than those related to tone and appeals; confrontational tone shows a negative association, though not statistically significant (Benjamini–Hochberg q-values reported across the 29 tests).

• Predictive accuracy: D + PW reaches 62.63% held-out accuracy on 800 held-out pairs, D_r + PW 62.13%, versus 63.75% for Tan et al.'s 3,205 hand-engineered features, 61.63% for 3,072-dimensional text-embedding-3-large embeddings, 59.25% for reply length alone, and 56.94% for zero-shot LLM judging (chance 50%).

• Compactness: The filtered set D_r has about half as many features as D yet loses at most half a point of accuracy; random subsets of D_r are more accurate than equally sized subsets of D by about one point on average over k ≤ 16 (paired 95% CIs [−0.1, 2.2] for PW and [0.0, 2.1] for BT).

• Expressiveness: Data-derived feature sets (S and D/D_r) are far less redundant than features elicited from the model's prior (P). At 16 features, D and D_r span about eight effective dimensions under Bradley–Terry scoring and six to seven under pointwise scoring, against about four for P.

• Data-blind features plateau: The prior-elicited set P is among the most predictive at low k but flattens quickly; the sample-elicited set S has more features (42) yet is least accurate among the expressive sets, sitting 4.2 points below D_r + PW at k = 16 and 3.6 points below on average (paired 95% CI [1.8, 5.4]).

Methodology in Plain English

The pipeline has four stages.

1. Collect rationales. Items are compared pairwise along a comparison graph. For each pair, an LLM judge is asked how the two items differ: it answers in free text, naming the distinguishing features and saying which item is stronger on each. These local calls need only a small context.

2. Canonicalize. Because the same quality is phrased differently across pairs, a second LLM call groups the raw descriptions into K shared features, yielding a sparse verdict tensor recording, for each pair and feature, whether the feature was cited and which item won on it.

3. Fit the choice model. A latent score matrix S (items × features) is estimated so that the LLM's behaviour is explained in two linked ways: bigger score gaps make a feature more likely to be cited, and, once cited, the higher-scoring item is more likely to be named the winner. Two nuisance terms absorb the judge's habits — a shared positional-bias term β (a preference for the first or second slot in the prompt) and per-feature baseline citation propensities γ_k (an eagerness or reluctance to mention a feature at all). Non-citation is treated as informative rather than missing. Estimation is penalized maximum likelihood with a ridge penalty λ‖S‖²_F, which also fixes the shift invariance of scores within a feature by selecting the centered representative. The result is per-item, per-feature scores with confidence intervals.

4. Filter and score new items. Features whose scores are poorly identified (high condition number κ_k) or noisy (low reliability ρ_k) are removed before a final refit. To score items not in the discovery set, the authors compare options: pointwise LLM scoring along the fixed features (one call per item, but subject to known pointwise biases), or pairwise comparison against a fixed number of already-scored items with a per-feature Bradley–Terry fit (keeps dataset-level aggregation and position-bias correction without re-running discovery). Full discovery with d comparisons per item costs d·n/2 LLM calls.

The judging models are GPT-5.4 (high reasoning) for discovery and GPT-5.4 mini for scoring, with text-embedding-3-large used to check whether a discovered feature's name aligns with a ground-truth dial's name. The paper frames the whole procedure as a way to orchestrate multiple LLM agents: a lead agent delegates comparisons to parallel sub-agents and recombines their rationales statistically. Code is released at a public GitHub repository.

Why This Matters

Research impact. The paper reframes LLM "rationales" — normally discarded or used as qualitative colour — as quantitative data for feature discovery, and gives the resulting estimator consistency guarantees despite a likelihood that is non-convex and a parameter count that grows with the number of items. It also shows that the benefit of multi-agent LLM setups depends on how work is split and recombined, not merely on using more agents.

Real-world applications.

• Automated essay and résumé grading with a transparent rubric that can be revised and audited rather than an opaque embedding score.
• Argumentation and platform analytics: quantifying which reply styles actually change minds in online debate, with named features such as evidence and cases versus tone and appeals.
• LLM evaluation and benchmarking: recovering the dimensions a judge actually rewards, and auditing the judge's own pairwise decisions via the decomposition in Appendix A.
• Compliance and high-stakes decision support where decision-makers must justify outcomes on human-readable criteria.

Industry relevance. The method offers a concrete prompt-and-aggregate recipe for teams that already use LLM-as-a-judge, along with explicit bias correction (position, citation propensity), cost accounting (d·n/2 calls versus one call per item), and confidence intervals on every score.

Future Directions

• Theory beyond the dense, single-feature case. Theorem 1 assumes a dense comparison design with a bounded number of comparisons per pair and K = 1, and the authors state it does not cover the sparse regime they optimize in practice; extending consistency to sparse graphs and multiple features is open.
• Adaptive comparison graphs. The graph is fixed in advance; choosing which pairs to compare next based on estimated uncertainty (and the diagnostics κ_k, ρ_k) could reduce the d·n/2 call cost.
• Better handling of the citation channel. γ_k absorbs feature-specific citation habits, but the paper does not claim the grouping step into K shared features is optimal; more principled canonicalization of free-text rationales is a natural next step.
• Evaluating the auditing extension. The decision-auditing framework in Appendix A is presented as a small extension without the large-scale empirical validation given to the discovery pipeline, so its usefulness for interpretable LLM scoring remains to be tested.

Target Audience

Researchers and practitioners working on interpretable machine learning, LLM-as-a-judge systems, and statistical models for pairwise comparison data; applied statisticians interested in M-estimation when the number of parameters grows with the sample; and product or policy teams who need auditable, rubric-based scoring of unstructured text rather than opaque embeddings. Readers should be comfortable with logistic/sigmoid models and basic maximum-likelihood estimation, but the paper's motivation and experimental results are accessible to a broader audience.

Authors’ abstract

Feature extraction from unstructured data typically relies on opaque embeddings unsuited to interpretable decision-making. Interpretable features, on the other hand, typically require costly, and often infeasible, hand-engineering. Large language models (LLMs) offer a natural bridge, as they can read raw data and describe it in interpretable terms. One could simply ask an LLM to extract features from a large sample of the dataset, but context limitations and diluted attention can lead to unsatisfactory results. We introduce a method that automatically discovers interpretable, decision-relevant features from unstructured data. Our key idea is to perform pairwise comparisons between items, asking an LLM to articulate, in interpretable terms, the features distinguishing each pair. We treat these explanations, or rationales, as data: we develop a statistical choice model that combines these rationales across the comparisons to identify relevant features and estimate per-item, per-feature scores. We first test our method on a synthetic hiring dataset of cover letters with known ground-truth features and show that it significantly outperforms direct LLM feature extraction in feature recovery. We then apply it to the ChangeMyView (Reddit) debate dataset, where it discovers several interpretable features associated with persuasion while reaching predictive accuracy comparable to established baselines. We also show how the same rationales can be used to audit LLM decisions, yielding a holistic score whose decomposition over interpretable features reveals what the LLM rewards.