SynQuE: Estimating Synthetic Dataset Quality Without Annotations

AI summary

Overview

Research area: Machine learning / data-centric AI — specifically synthetic training data selection and dataset quality estimation under label scarcity (the paper's task domains are sentiment analysis, Text2SQL, web navigation, and image classification).

Technical level: Intermediate. The paper assumes familiarity with embedding models, distribution-divergence metrics, and fine-tuning, but its framing of the problem is accessible.

Scope: The paper defines the Synthetic Dataset Quality Estimation (SynQuE) problem — ranking candidate synthetic datasets by their expected real-world task performance using only unannotated real samples — establishes benchmarks for it, and introduces a suite of proxy metrics including a new LLM-reasoning-based proxy called Lens.

What This Paper Is About

Generative models can produce large annotated synthetic datasets for tasks where real data is scarce because it is expensive or privacy-sensitive to collect, but the quality of those synthetic datasets varies widely, and bigger synthetic datasets do not always help — the paper notes they can even degrade performance, exhibiting inverse scaling. The core problem is that deciding which synthetic dataset to train on normally requires labeled real validation data and training a model on every candidate dataset, both of which are unavailable in "cold-start" settings. SynQuE asks whether a proxy score computed from a synthetic dataset plus a small set of unannotated real inputs can rank synthetic datasets so that the top-ranked ones actually produce the best real-world task performance.

Key Contributions

1. Formalizes the SynQuE problem and builds the first comprehensive benchmark for it. The paper defines quality as: synthetic dataset A is better than B if a model trained on A performs better on a real-world test set (Eq. 1), and shows this ideal is infeasible without labeled real data and repeated model training, so it substitutes proxy metrics Q(D_s^(k), U_r) that use only synthetic data and unannotated real samples.
2. Introduces the first SynQuE proxy metrics by adapting distribution and diversity measures to this setting through embedding models: mean distance to medoids (MDM), Maximum Mean Discrepancy (MMD²), Proxy-A-Distance (PAD), and the generative baseline Mauve — none of which had been systematically evaluated for synthetic data selection.
3. Proposes Lens (LLM-Evaluated Normalized Score), a proxy that uses LLM reasoning to compile natural-language "rubrics" describing how synthetic data differs from real data, then has an LLM score how likely each synthetic example is to belong to the real dataset, with a debiasing scheme covering order bias, label bias, and score bias.
4. Runs a comprehensive evaluation across four domains — sentiment analysis, Text2SQL, image classification, and web navigation — measuring how well each proxy correlates with and selects for real task performance, with five random seeds for the real-sample subsets.

Main Findings

• Proxy metrics do correlate with real task performance. Table 3 reports moderate to strong Spearman and Pearson correlations between SynQuE proxies and downstream performance across the four task domains, and the paper states that the best proxy varies by task.
• Selecting top-3 datasets by proxy beats indiscriminate selection. In Table 2, "mean" performance (averaging all synthetic datasets) is compared against the top-3 datasets chosen by each proxy. On Text2SQL Apps, mean performance is 30.4, and the paper's abstract reports that top-3 selection via SynQuE proxies raises accuracy to 38.4 (+8.1)% on average compared to selecting data indiscriminately. On Text2SQL overall, the mean across the three BIRD databases is 37.8, with the highest reported proxy average reaching 43.8 (+6.0).
• Lens 32B (debiased) is claimed to be the only proxy with consistently positive correlation and selection gains across all tasks and splits. The paper attributes this to Lens capturing nuanced task details — for example, a rubric point distinguishing that one website's dataset focuses on data retrieval (temperature anomalies, moon phases) while the other emphasizes applied computational problem-solving.
• Lens is strongest on complex, long-horizon tasks. The advantage is most pronounced in web navigation, where representation-based metrics struggle, because Lens reasons over rich structured inputs such as accessibility trees that dense embeddings compress poorly.
• Image classification is the hard case. The synthetic image data (built with Stable Diffusion 1.1 and 1.5) contains visual variability and label ambiguity — especially in Split 2 between classes like "stage" and "throne" — producing inconsistent correlations across splits. The paper hypothesizes VLM rubric generation fails to capture meaning from batches of images, and expects this to improve as VLMs improve.
• More unannotated real data improves reliability. Table 6 shows that increasing the number of real samples from |U_r| = 25 to |U_r| = 50 consistently strengthens correlations for all proxy metrics (Table 3 uses 30 real samples).
• Perplexity-based scaling approaches fail in this setting. As shown in Table 5, a perplexity proxy on BIRD Text2SQL has correlations of −.31/−.33 (Computers), −.25/−.29 (Apps), .24/.31 (Movies), averaging

Authors’ abstract

We introduce and formalize the Synthetic Dataset Quality Estimation (SynQuE) problem: ranking synthetic datasets by their expected real-world task performance using only limited unannotated real data. This addresses a critical and open challenge where data is scarce due to collection costs or privacy constraints. We establish the first comprehensive benchmarks for this problem by introducing and evaluating proxy metrics that choose synthetic data for training to maximize task performance on real data. We introduce the first proxy metrics for SynQuE by adapting distribution and diversity-based distance measures to our context via embedding models. To address the shortcomings of these metrics on complex planning tasks, we propose LENS, a novel proxy that leverages large language model reasoning. Our results show that SynQuE proxies correlate with real task performance across diverse tasks, including sentiment analysis, Text2SQL, web navigation, and image classification, with LENS consistently outperforming others on complex tasks by capturing nuanced characteristics. For instance, on text-to-SQL parsing, training on the top-3 synthetic datasets selected via SynQuE proxies can raise accuracy from 30.4% to 38.4 (+8.1)% on average compared to selecting data indiscriminately. This work establishes SynQuE as a practical framework for synthetic data selection under real-data scarcity and motivates future research on foundation model-based data characterization and fine-grained data selection. We release our code.