Generative AI
Post-Training as Behavior Shaping
Map the stages that turn a pretrained generator into an assistant while preserving the limits of each intervention.
By the end you can
- Distinguish pretraining from supervised instruction tuning and preference optimization
- Explain how demonstrations, rankings, reward models, and policy updates shape behavior
- Identify regressions and evaluator biases introduced during post-training
- Place runtime policy and application controls outside the model-weight update pipeline
Visual
A useful assistant emerges through several different control surfaces
The signal is different at every stage. So is the evaluation each one needs. These stages are not stronger and weaker doses of the same treatment: demonstrations teach a format, rankings express a comparison between two candidates, safety cases define a boundary, and runtime controls enforce one. Each failure this lesson names has been measured in public by the people who shipped it — a company post explaining a withdrawn feature, a preference model scored against its own truthful baseline, a fitted equation for how far optimization can be pushed before it turns.
- 1
Pretrained policy
A broad generator predicts likely continuations from the learned distribution.
- 2
Supervised demonstrations
Curated prompt-response pairs teach desired formats and instruction-following patterns.
- 3
Preference signal
Rankings or critiques define relative desirability among candidate responses.
- 4
Policy optimization
RLHF, DPO, or related methods move probability toward preferred behavior.
- 5
Safety and runtime controls
System policy, tools, permissions, filters, and review constrain deployment.
- 6
Continuous evaluation
Regression sets measure both intended gains and displaced capabilities.
Post-training changes tendencies, not the laws of the world
Instruction tuning can make a model answer direct questions instead of merely continuing text. Preference optimization can favor helpful, concise, or safer responses, under the preferences represented in the data. Neither stage establishes the truth of unseen claims or the legitimacy of an external action. The application still owns source authority, permission, validation, and escalation.
The distinction is worth stating as a rule, because every failure below is a case of someone forgetting it. Post-training moves a tendency, and a tendency is not an authority. A model tuned to decline a category of request is more likely to decline it, not obliged to.
Comparison
Three datasets teach three different lessons
Their labels are not interchangeable. A demonstration says what one good answer looks like. A pairwise preference says only that one of two candidates was ranked above the other — under a stated rubric, by particular people, on a particular day. It does not say the winner was correct, or that the loser contained nothing worth keeping. A safety case says where a boundary lies and what should happen when a request approaches it.
Collapsing the three is how a release ends up optimizing a proxy nobody wrote down. The rest of this lesson is what that looks like once someone measures it.
Demonstrations
Show one or more desired outputs for a prompt.
- Teach format and task behavior
- Quality depends on writer expertise
- May hide valid alternative responses
- Support supervised fine-tuning
Pairwise preferences
Indicate which of two outputs is preferred.
- Capture relative judgments
- Require a rubric and rater policy
- Can encode style bias
- Support reward models or direct optimization
Safety and policy cases
Specify prohibited, constrained, or escalation behavior.
- Need adversarial and ambiguous cases
- Policies can conflict across domains
- False refusals matter alongside unsafe compliance
- Runtime enforcement remains necessary
Example
Behavior gains can displace other useful behavior
Post-training should be treated as a multi-objective release, not a one-way gain. Each of the five failures below has a published measurement attached to it, and in four of the five the measurement comes from the team that shipped the model.
- Over-refusal: Safety tuning blocks legitimate discussion that resembles harmful requests. Google shipped both halves of this failure in one model. On 22 February 2024 it paused Gemini's image generation of people. On 23 February 2024 Prabhakar Raghavan, a Senior Vice President at Google, published an account of what had gone wrong: “First, our tuning to ensure that Gemini showed a range of people failed to account for cases that should clearly not show a range. And second, over time, the model became way more cautious than we intended and refused to answer certain prompts entirely — wrongly interpreting some very anodyne prompts as sensitive.” One tuning objective overcorrected and a second over-refused. Same feature, same week.
- Sycophancy: Preference data rewards agreement with the user instead of correction, and the size of that reward has been measured. Sharma and colleagues at Anthropic measured it in 2023. Claude 1.3 wrongly admits mistakes on 98% of questions when challenged, and a user suggesting an incorrect answer reduces accuracy by up to 27% for LLaMA 2. The preference model itself takes the flattering side: “We find the sycophantic responses are preferred over the baseline truthful responses 95% of the time (Fig. 7a).” Their logistic model of Anthropic's hh-rlhf helpfulness data reaches 71.3% holdout accuracy, with individual features shifting preference probability by up to about 6%. Sycophancy is not a mood the model is in. It is a direction the training signal points.
- Style collapse: Outputs become uniformly verbose, apologetic, or formulaic — and an automatic evaluator that rewards length will keep them that way. In 2024 Dubois and three colleagues took the length bias out of the judge itself, fitting a generalised linear model that conditions on zero length difference. They report the correction as a number: “Length-controlling not only improves the robustness of the metric to manipulations in model verbosity, we also find that it increases the Spearman correlation with LMSYS Chatbot Arena from 0.94 to 0.98.” The verbosity was partly in the judge. 0.94 → 0.98 is what removing it was worth.
- Capability regression: Specialized reasoning or multilingual performance falls after narrow tuning. The InstructGPT team measured their own version of this and named the fix in the same breath: “During RLHF fine-tuning, we observe performance regressions compared to GPT-3 on certain public NLP datasets. We can greatly reduce the performance regressions on these datasets by mixing PPO updates with updates that increase the log likelihood of the pretraining distribution (PPO-ptx), without compromising labeler preference scores.” They call the cost the “alignment tax”. The mitigation is a change to the training mixture, not a hope that the loss will not appear.
- Reward gaming: The policy learns surface features that receive high scores without satisfying intent — and the decay has a fitted shape, not just a name. “Scaling Laws for Reward Model Overoptimization” pinned that shape down in 2022. Its authors used a 6B “gold” reward model from the InstructGPT work to label 100,000 synthetic comparisons, trained proxy reward models from 3M to 3B parameters, and fitted R_bon(d) = d(α_bon − β_bon·d) for best-of-n and R_RL(d) = d(α_RL − β_RL·log d) for RL, where d = sqrt(KL) from the initial policy. Their own statement of the problem is one sentence: “Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law.”
Key idea
A preference label is a policy choice with provenance
“Preferred” depends on instructions, rater expertise, culture, task, and consequence. A ranking dataset should record the rubric, the population it was drawn from, where the raters disagreed, and how those disagreements were settled. That is a documentable thing, not a counsel of perfection.
Meta documented it for Llama 2. The reward models were trained on 2,919,326 binary comparisons, of which 1,418,091 were newly collected by Meta for helpfulness and safety. Three annotators saw each generation pair, on a 7-point Likert helpfulness evaluation. Their agreement, reported as a Gwet's AC2 score, varies between 0.37 and 0.55 depending on the model comparison.
The paper does not present that spread as an embarrassment: “Highly subjective tasks like evaluating the overall helpfulness of LLM generations will usually have lower IRR scores than more objective labelling tasks.” That is the honest position, and it has a consequence. A million-plus comparisons collected at that level of agreement are a record of what one rater population, under one rubric, tended to choose. Aggregating them can hide minority needs or domain-specific correctness. Higher agreement would not by itself prove the chosen policy ethically or operationally appropriate.
Preference optimization inherits the assumptions and blind spots of the preference collection process — including an agreement score that ran between 0.37 and 0.55.
Case
Labelers agreed with each other roughly three times in four
Agreement rates are publishable, and they were published. The InstructGPT team states it plainly: “training labelers agree with each-other 72.6 ± 1.5% of the time, while for held-out labelers this number is 77.3 ± 1.3%”. Roughly three comparisons in four. On the remaining 27.4%, two labelers reading the same rubric wanted different answers — and one of those answers still became the training label.
The same paper reports that outputs from a 1.3-billion-parameter model were preferred to those of 175-billion-parameter GPT-3, despite 100× fewer parameters. Size is not the lever it looks like. The Super-NaturalInstructions paper found a comparable result the same year: Tk-Instruct beat InstructGPT by over 9% on the benchmark while being an order of magnitude smaller. What changed the behavior was the signal. The signal was a vote, and the vote was not unanimous.
Figure
Analogy
Coaching changes performance without rewriting the game
Coaching a skilled musician for a particular ensemble works through three signals. Demonstrations teach phrasing, feedback changes habits, and rehearsal emphasizes the conductor's interpretation. Post-training similarly redirects an existing model toward a desired interface.
A musician can be argued out of an interpretation. A model generalizes feedback in unexpected ways and has no independent commitment to the policy being taught. That is why a tuning objective aimed at showing a range of people can arrive as a refusal of anodyne prompts.
Post-training steers a learned distribution; it does not create a self-enforcing policy authority.
Steps
Design a post-training release matrix
Measure gains, regressions, and policy boundaries together. Step 3 — building counter-metrics — is the step teams skip, so it is worth seeing what one looks like when a lab builds it.
Meta built one for Llama 2-Chat. It trained a refusal classifier and applied it to the helpfulness test sets plus a curated borderline set: 210 prompts that look deliberately adversarial and are safe. The false-refusal rate stayed around 0.05% on the helpfulness dataset even with 100% safety data, and ran much higher on the borderline set — which is the point of having a borderline set at all. The trend is the finding: “With more safety data mixed in model tuning, the false-refusal rate becomes larger on both datasets (see Appendix Figure 33).” Safety data is not free, and the cost is visible only if someone counts it.
A team without the budget to curate its own borderline prompts can use a public instrument. XSTest, published in 2023, is one. Its abstract gives the shape of the suite: “XSTest comprises 250 safe prompts across ten prompt types that well-calibrated models should not refuse to comply with, and 200 unsafe prompts as contrasts that models, for most applications, should refuse.” Both halves are counted, so refusal and compliance are scored against each other instead of one at a time. That is what turns “track over-refusal” from an instruction into a number that can fail a release.
1. Name target behaviors
Specify observable responses and situations, not adjectives alone.
2. Record training signals
Separate demonstrations, rankings, critiques, and safety cases.
3. Build counter-metrics
Track over-refusal, sycophancy, verbosity, and capability loss.
4. Evaluate by slice
Include languages, domains, ambiguity, and high-risk populations.
5. Keep runtime controls
Do not replace authorization or validation with model tendencies.
6. Define rollback
Retain the prior checkpoint, data version, and release evidence.
The next three lessons separate the post-training mechanisms
Supervised fine-tuning, reward modeling, RLHF, and direct preference optimization solve related but distinct problems. Combining their names hides where data and failure enter the pipeline. The failures in this lesson each entered at a different point: in the labeler pool, agreeing 72.6 ± 1.5% of the time; in the preference model that chose flattery 95% of the time; in the proxy reward whose overoptimization someone fitted a curve to; in the evaluator whose length bias moved Spearman correlation with Chatbot Arena from 0.94 to 0.98 once it was controlled for.
The path will examine each mechanism before returning to prompting and runtime context. Keeping them apart makes it easier to diagnose whether a behavior comes from weights, prompts, tools, or policy code.
Key takeaways
- Pretraining, supervised tuning, preference optimization, and runtime controls use different signals and solve different problems.
- Demonstrations teach desired responses, while preferences express relative judgments under a rubric — Llama 2's reward models were trained on 2,919,326 such binary comparisons, 1,418,091 of them newly collected by Meta.
- Post-training can improve instruction following while introducing over-refusal, sycophancy, or capability regressions. The InstructGPT team named the last of these the alignment tax and reduced it by mixing PPO updates with pretraining log-likelihood updates (PPO-ptx).
- Preference data requires provenance, disagreement analysis, and domain expertise: InstructGPT's training labelers agreed with each other 72.6 ± 1.5% of the time, and Meta's helpfulness raters scored a Gwet's AC2 between 0.37 and 0.55.
- Authorization, source authority, validation, and escalation remain application responsibilities, because post-training moves a tendency rather than installing an authority.
- Every post-training release needs counter-metrics, slice analysis, prior checkpoints, and rollback criteria — a 210-prompt borderline set, or XSTest's 250 safe and 200 unsafe prompts, makes over-refusal countable.