Synthetic Data and Model-Generated Supervision

A model that grades its own writing gives it better marks. Two summarization tasks, three graders: GPT-3.5 Turbo, GPT-4 and Llama 2 each scored their own summaries above other summaries that human annotators had rated of equal quality. Panickssery and two co-authors published the measurement at NeurIPS in 2024.

The mechanism is recognition. “GPT-4 is 73.5% accurate at distinguishing its outputs from those of two other LLMs and humans.” The judge knows its own handwriting, and nobody trained it to. GPT-3.5 and Llama 2 passed 90% self-recognition accuracy after fine-tuning on 500 examples. Self-preference strength rose linearly with self-recognition ability.

So a team generates fifty thousand support conversations with one strong model, fine-tunes a smaller model on them, then grades the result with prompts from the same model family. It is not measuring quality. It is measuring family resemblance, and the offline scores will rise.

The validation burden depends on what the synthetic record claims to represent. Input augmentation — new prompts, paraphrases, rare scenario combinations — expands linguistic and scenario coverage, but it repeats generator style. It stands or falls on deduplication and diversity checks. Target generation — candidate answers, rationales, structured outputs — scales demonstration production and requires strong semantic review, because a fabricated fact arrives in the same register as a correct one. It is defensible where validators exist. Adversarial generation — prompt injection, ambiguity, unsafe requests, edge cases — widens stress-test coverage, but it misses attacks the generator has never seen. So it belongs beside a human red team, and the evaluation cases it produces must stay protected.

Preference and critique generation is the one role with a published head-to-head behind it. Google ran the comparison and reported it at ICML in 2024. Human evaluators preferred the policy trained on model-written preferences over the supervised fine-tuned baseline 71% of the time on summarization, against 73% for the policy trained on human preferences. On helpful dialogue the two figures were 63% and 64%. Neither difference was statistically significant, and head-to-head, RLAIF against RLHF was indistinguishable from a 50% win rate.

On harmless dialogue the model-written labels did better than the human ones: “For harmless dialogue generation, RLAIF achieves a harmless rate of 88%, outperforming both RLHF and SFT, which score 76% and 64%, respectively.”

Cheaper labels can match expensive ones on some axes. That is a measured result on three tasks, not a general licence to stop asking humans.

The generator should not be the sole authority for its own outputs. The path runs like this. Define the missing coverage: tasks, slices, difficulty or attacks absent from real data. Generate with variation — multiple prompts, models, seeds and controlled scenario variables. Apply deterministic filters for schema, duplicates, policy constraints, calculations and source references. Review a risk-based sample, using experts for consequential content and disagreement cases. Measure diversity and realism against the real data's style, length, topics, errors and feature distributions. Approve only when the target production behavior improves on real held-out data.

The fourth step is the one teams try to automate away, and there is a measurement of how far that can go. Zheng and colleagues put GPT-4 in the judge's seat and checked it against people, at NeurIPS in 2023: “Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.” The released evidence base is 3K expert votes and 30K conversations with human preferences. The same paper documents position, verbosity and self-enhancement biases in LLM judges.

Both halves of that finding are load-bearing. A model judge agrees with people about as often as people agree with each other, which is why an automated acceptance path is worth building. It is also wrong in patterned, nameable ways. That is what the human sample is for: the pattern is the thing it catches.

Compare the dataset's claims and distribution with the evidence it is meant to supplement. Record generation lineage: model, prompt, seed, source inputs, filters, reviewer. Detect duplicates and templates by measuring lexical, semantic and structural repetition. Validate labels and facts with tools, sources, experts and cross-model disagreement. Compare vocabulary, difficulty, error types and population slices with the real data. Run ablations that train with and without the synthetic subsets under matched budgets. Keep real release cases outside the generation and selection loops.

Step 2 is where teams usually settle for an impression, and Self-Instruct shows what a published threshold looks like instead. Wang and colleagues did not inspect their pool for variety. They wrote the rule down, in the paper's “Filtering and Postprocessing” paragraph: “To encourage diversity, a new instruction is added to the task pool only when its ROUGE-L similarity with any existing instruction is less than 0.7.” Instructions containing keywords the model could not process — image, picture, graph — were excluded outright. What survived that filter was the 52,445-instruction pool: 11,584 classification tasks and 40,861 that were not.

A number someone else can recompute on your dataset is worth more than a paragraph asserting that the examples are varied. “Measure lexical repetition” is an activity. ROUGE-L below 0.7 against every instruction already in the pool is a gate.

Generated examples can produce training data faster, expose edge cases, and teach smaller models. That last claim has a worked case with numbers attached. phi-1 is a 1.3B-parameter Transformer code model, trained for 4 days on 8 A100s. Its diet was 6B tokens of filtered “textbook quality” web data plus 1B tokens of textbooks and exercises generated synthetically with GPT-3.5. Microsoft Research published it in 2023. The outcome: “Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP.” A 350M-parameter variant, phi-1-small, run through the same pipeline, still scored 45% on HumanEval. What carried those results was the curation of the 1B synthetic tokens, not their quantity.

Set the two poles of this lesson side by side. Nine unfiltered generations of OPT-125m ended in jackrabbit nonsense. 1B curated synthetic tokens put a 1.3B-parameter model at 50.6% pass@1. The value of generated data depends on independent validators and real held-out gains, and that is the only difference between the two.

The next module moves into multimodal systems. Synthetic media increases both capability and provenance risk when generated artifacts resemble evidence from the world.