Instruction Tuning and Supervised Fine-Tuning

Fine-tune a small model on ChatGPT's outputs and it starts to sound like ChatGPT. Whether it also works like ChatGPT is a separate question, and it has been measured. Imitation models of 1.5B to 13B parameters were trained on 0.3M to 150M tokens of ChatGPT outputs, then put in front of crowd raters. The raters judged them competitive with ChatGPT itself.

Targeted automatic evaluations did not agree. On tasks not heavily supported in the imitation data, the models closed little to none of the gap from the base LM to ChatGPT. Gudibande and colleagues state the mechanism in one line of their abstract: "We show that these performance discrepancies may slip past human raters because imitation models are adept at mimicking ChatGPT's style but not its factuality." The paper is The False Promise of Imitating Proprietary LLMs, an ICLR 2024 spotlight.

The demonstrations taught the surface of the behavior. The human evaluation could not tell the surface from the task. Both facts were measured, not suspected.

Supervised fine-tuning continues gradient-based training on curated input-output pairs. For a decoder model, the loss often applies to response tokens while the prompt provides conditioning context. The model learns both the task and the dataset's presentation habits. A narrow demonstration set can improve one interface while making responses more repetitive or brittle elsewhere.

The order of magnitude is smaller than most people expect. InstructGPT was built on about thirteen thousand demonstrations. Ouyang and co-authors set the three corpora out in 2022: "The SFT dataset contains about 13k training prompts (from the API and labeler-written), the RM dataset has 33k training prompts (from the API and labeler-written), and the PPO dataset has 31k training prompts (only from the API)."

Those roughly 13,000 demonstrations were written by a hired team of about 40 contractors. What they bought was not marginal. Human raters preferred the 1.3B-parameter InstructGPT's outputs to those of the 175B GPT-3, despite the 100x difference in parameters. Demonstrations are a lever that a hundredfold in scale did not match.

A usable demonstration record carries five layers, and only the middle one is the answer. Task intent records what user need and decision the example represents. Input and context hold the prompt, source material, tool state and relevant constraints. The target response is the desired output with its exact formatting and behavior. Provenance names the author, source, policy version, creation method and review history. Evaluation tags mark domain, difficulty, risk, language and known alternatives. The metadata is what makes later audits and corrections possible. Without it, a dataset can only be regenerated, not repaired.

A public corpus shows what this costs and what it yields. Köpf and co-authors released OpenAssistant Conversations in 2023, and their abstract counts it out: "In an effort to democratize research on large-scale alignment, we release OpenAssistant Conversations, a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 complete and fully annotated conversation trees." It came from over 13,500 volunteers, under a permissive licence.

Note the ratio: 461,292 quality ratings against 161,443 messages. In a corpus built to be audited, the annotation outweighs the content.

A healthy mixture uses each source for the behavior it can support.

Expert-written examples are high-control demonstrations created for the intended policy. They come with clear rationale and provenance, they are expensive and slow to scale, they can overrepresent idealized prompts, and they need disagreement and adjudication. InstructGPT's roughly 13,000 SFT prompts, written with labeler demonstrations by about 40 contractors, are what this source looks like at production scale.

Historical interaction logs supply real wording and operational diversity from deployed workflows. They capture authentic user behavior, they contain stale policy and privacy risk, and they reflect prior system selection effects. They require redaction and reconstruction before any of that authenticity is usable.

Synthetic examples are model-generated prompts or answers reviewed before use, and their value has been measured. Self-Instruct bootstrapped GPT-3 from 175 seed instructions into 52K instructions and 82K instances. A candidate was admitted only when its ROUGE-L similarity to every existing instruction was below 0.7. Wang and co-authors report the result in their abstract: "Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations." The remaining distance to that human-annotated baseline was 5% absolute.

So synthetic generation expands rare combinations quickly and cheaply. It needs an explicit near-duplicate filter to stay usable at all. And it can reproduce the generator's blind spots — which is exactly what Gudibande and colleagues measured when imitation of ChatGPT outputs transferred style without factuality. Synthetic data earns a place in the mixture. It should not dominate the evidence base.

Treat the dataset as a versioned product artifact. Step 1, define behaviors and exclusions: write what the model should do, refuse, clarify and escalate. Step 2, sample the real distribution, covering ordinary, rare, adversarial and ambiguous requests. Step 3, create and review targets with domain experts, recording acceptable alternatives. Step 4, deduplicate and split so that near-duplicate examples cannot leak into evaluation. Step 5, verify formatting and masks by testing tokenization, role boundaries, truncation and loss application. Step 6, run broad regressions measuring target gains, style drift, multilingual behavior and safety tradeoffs.

Steps 2 and 4 are not aspirational. Ouyang and co-authors documented exactly these controls for InstructGPT: "We heuristically deduplicate prompts by checking for prompts that share a long common prefix, and we limit the number of prompts to 200 per user ID. We also create our train, validation, and test splits based on user ID, so that the validation and test sets contain no data from users whose data is in the training set. To avoid the models learning potentially sensitive customer details, we filter all prompts in the training split for personally identifiable information (PII)." Deduplication, a per-user cap, user-level splitting and PII filtering — written down as part of the dataset description.

The cost of skipping step 4 has been quantified. C4 contained a single 61-word English sentence repeated over 60,000 times. Over 1% of unprompted model output was copied verbatim from training data. Deduplication cut memorized emissions tenfold. On the evaluation side, Lee and co-authors are specific: "We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation." Over 4% of a validation set is a number large enough to decide a launch.

Step 6 is where fine-tuning stops being a local edit. Qi and co-authors report: "For instance, we jailbreak GPT-3.5 Turbo's safety guardrails by fine-tuning it on only 10 such examples at a cost of less than $0.20 via OpenAI's APIs, making the model responsive to nearly any harmful instructions." Ten examples and twenty cents. The result that matters more for ordinary teams is the second one: fine-tuning on benign, commonly used datasets also degraded safety alignment, with no malicious intent anywhere in the process. A regression suite is what stands between an ordinary SFT run and that outcome.

Supervised examples work well for formats, procedures, tone and task-specific response patterns. Those are the properties that transferred cleanly in every study above, including in the imitation models whose style convinced human raters. They are weaker when the desired behavior depends on unstable facts, or on preferences that are difficult to express as one ideal answer. That is where the same imitation models closed little to none of the gap, and where a benign dataset can quietly move safety behavior.

The next lesson examines preference data and reward models. Those methods compare alternatives instead of requiring a single canonical response. Note that in InstructGPT the reward-model corpus was the larger of the two: 33k training prompts against about 13k demonstrations.