Advanced techniques
Few-Shot and In-Context Learning
Separate metric-based few-shot learning, optimization-based adaptation, and in-context behavior in pretrained language models.
By the end you can
- Distinguish few-shot fine-tuning, episodic meta-learning, metric-based classification, and in-context learning
- Explain the roles of support sets, query sets, demonstrations, and task distributions
- Design evaluations that measure sensitivity to example choice and order
- Recognize when retrieval, fine-tuning, or ordinary supervised learning is a better option
Comparison
Four meanings of “few-shot”
The term describes several different learning settings that should not be evaluated as if they were identical. Few-shot supervised fine-tuning updates model parameters on a very small labelled target set. Metric-based few-shot learning classifies queries by comparing them with a small labelled support set, so new classes can appear at test time. Optimization-based meta-learning learns an initialization or an update rule that adapts quickly, using an inner and an outer loop. In-context learning conditions a pretrained model on instructions and demonstrations, with no task-specific gradient step at inference at all.
The third of those comes with a published headline number and a published problem. It is worth seeing the two together. MAML was reported in 2017 at 48.70 ± 1.84% on 5-way 1-shot miniImageNet and 63.11 ± 0.92% on 5-shot. Antoniou and colleagues reproduced the method independently two years later and measured 48.25 ± 0.62% and 64.39 ± 0.31%. The 1-shot figure lands within half a point of the original, which is a good result for a reproduction.
What they documented alongside that agreement matters more than the agreement itself: “Depending on the neural network architecture and the overall hyperparameter setup, MAML can be very unstable during training as illustrated in Figure 1.” The instability was severe enough to motivate a set of training-procedure fixes. Their repaired variant, MAML++, reached 52.15 ± 0.26% and 68.32 ± 0.44%.
So the honest label on the optimization-based card is not that such methods can be compute intensive. It is that the published accuracy replicates, and that the training procedure needed architecture-specific repair before anyone should rely on reproducing it.
Few-shot supervised fine-tuning
Update model parameters using a very small labeled target set.
- Uses gradient updates
- High overfitting risk
- Can exploit pretrained features
- Needs repeated splits or episodes
Metric-based few-shot learning
Classify queries by comparing them with a small labeled support set.
- Often learns an embedding space
- Supports new classes at test time
- Depends on episode design
- Examples: matching and prototypical networks
Optimization-based meta-learning
Learn an initialization or update rule that adapts quickly.
- Uses inner and outer learning loops
- Targets rapid parameter adaptation
- Can be compute intensive
- Example: MAML-style methods
In-context learning
Condition a pretrained model on instructions and demonstrations without parameter updates.
- Behavior changes through the prompt context
- Sensitive to wording, order, and examples
- No task-specific gradient step at inference
- Most associated with large language models
The task distribution matters more than the headline shot count
A five-shot learner is not defined only by seeing five examples. It is defined by what kinds of tasks, classes, and support–query relationships it encountered during development. Episodic training samples small tasks so that the training procedure resembles the intended adaptation setting. If training episodes are much easier or structurally different from deployment, the few-shot claim may not transfer.
The cleanest demonstration of that is an algorithm compared with itself. Prototypical networks were trained with 30-way episodes for 1-shot and 20-way episodes for 5-shot, and reported 49.42 ± 0.78% and 68.20 ± 0.66% on miniImageNet. Snell and colleagues were explicit about why: “We have found, however, that it can be extremely beneficial to train with a higher Nc, or “way”, than will be used at test-time.” Chen and colleagues later re-ran the same method inside a single codebase under a uniform 5-way meta-training protocol and measured 44.42 ± 0.84% and 64.24 ± 0.72%. When they reproduced the higher-way episodes instead, they got 47.74 ± 0.84% and 66.68 ± 0.68%. Same algorithm, same benchmark, same shot count at test time. A swing of 2.4 to 3.3 points was produced only by how the training episodes were composed.
For metric-based classification, the support set defines the temporary classes. The model maps support and query examples into a representation space, then predicts by similarity or class prototypes. For in-context learning, demonstrations communicate the task through the input sequence, and the model may rely on semantic cues, formatting, label tokens, or memorized patterns rather than a single clean mechanism. Whoever tests the system should therefore vary the support examples, order, label names, instructions, and task families. Reporting one hand-picked prompt or one lucky support set overstates reliability.
The bars above are GPT-3, “an autoregressive language model with 175 billion parameters”. It reached “64.3% accuracy on TriviaQA in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting”, and the paper notes that the few-shot figure “is state-of-the-art relative to fine-tuned models operating in the same closed-book setting”. No gradient update produced any of that gain. The task and the demonstrations were “specified purely via text interaction with the model”.
Figure
Few-shot performance is a distribution over tasks and demonstrations, not a property of one prompt.
Visual
An episodic evaluation unit
The episode is the unit that simulates rapid adaptation to a new task: sample a task, build the support set, adapt or condition, evaluate on held-out queries, repeat across many tasks and report the spread.
Step one is not the neutral bookkeeping it looks like. It is where the 68.20 ± 0.66% and the 64.24 ± 0.72% above were decided. Choosing 30-way or 20-way training episodes rather than a uniform 5-way moved prototypical-network accuracy by 2.4 to 3.3 points, while the test-time episodes stayed identical. A comparison between two published few-shot numbers is therefore partly a comparison between two episode samplers. A paper that does not state its meta-training way has not stated its method.
- 1
Sample a task
Choose classes, a user, a domain, or a small target problem.
- 2
Build the support set
Provide the limited labeled examples available for adaptation.
- 3
Adapt or condition
Update parameters, compute prototypes, or construct a prompt.
- 4
Evaluate on queries
Measure performance on unseen examples from the same episode.
- 5
Repeat across tasks
Report mean, spread, and failure patterns across many episodes.
A reliable result averages over many new-task simulations.
Example
What makes few-shot results unstable
Small changes can have large effects because each example carries unusual weight. Each of these has been measured rather than merely feared.
- Support-set composition: One atypical demonstration can redefine the temporary decision boundary, and with five labelled examples there is nothing in the set to average it away.
- Example order: In-context models respond differently when the same demonstrations are rearranged. Lu and colleagues measured the range on a fixed demonstration set and recovered 13% relative with an entropy-based ordering — the next section gives the numbers.
- Label wording: Whether label semantics bind at all depends on scale. With 100% of the labels flipped, text-davinci-002 fell from 90.3% to 22.5% — far below chance, because it followed the contradicted labels. Smaller models stayed near their unflipped performance.
- Task mismatch: Meta-training episodes may not represent deployment tasks, and the penalty is quantified. On the cross-domain mini-ImageNet→CUB transfer with a ResNet-18 backbone, Chen and colleagues recorded 51.34 ± 0.72% for MAML against 65.57 ± 0.70% for a plain fine-tuned baseline.
- Class imbalance: A tiny support set can accidentally omit important within-class variation, and no amount of averaging over queries recovers a class the support set under-represents.
- Prompt formatting: Delimiters, output schemas and instructions become unintended shortcuts. The size of the effect rivals the choice of model: “We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt formatting in few-shot settings, with performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B.”
Case
The same demonstrations, reordered, from near state of the art to chance
Take a fixed set of demonstrations and permute them. Same examples, same labels, same model, same count — only the sequence changes. Lu and colleagues did exactly that. Order alone “can make the difference between near state-of-the-art and random guess performance”, and the sensitivity “is present across model sizes”, so it is not a defect that scale quietly removes. Nor can a fix be borrowed: “a given good permutation for one model is not transferable to another”. Their entropy-based selection over candidate orderings “yields a 13% relative improvement for GPT-family models across eleven different established text classification tasks”.
Set that beside the TriviaQA panel in the previous section and the shape of the problem is visible in one comparison. Adding demonstrations to the prompt was worth 6.9 points there — 64.3 to 71.2. Rearranging demonstrations, elsewhere, can be worth the distance between a leaderboard result and guessing. The two measurements come from different benchmarks and different baselines, and do not net against one another. They say one thing: the arrangement of the prompt is not a smaller variable than its contents.
Analogy
Learning a board game from three demonstrations
Three rounds of an unfamiliar board game are enough to infer the objective, and enough to copy superficial moves or misunderstand a rare exception. A fourth example that exposes the exception could radically change your rule. Few-shot systems face the same ambiguity, because limited evidence underdetermines the task.
A pretrained model adds an important twist. It arrives with extensive prior experience, so its behaviour reflects both the demonstrations and earlier patterns. The two can be separated experimentally rather than argued about. Min and colleagues replaced the gold labels in the demonstrations with random ones and re-ran twelve models, GPT-3 among them: “We then find that replacing gold labels with random labels only marginally hurts performance.” The cost was 0–5% absolute, averaging 1.7% on multi-choice tasks and 2.6% on classification.
If demonstrations taught the task the way labelled training data teaches a classifier, corrupting every label could not cost so little. What they mainly convey is the shape of the task — the input format, the label space, the kind of answer expected. The board-game analogy holds only if the player already knows a hundred board games and is largely being told which one this is. That is why the prior has to be tested rather than ignored. Whoever evaluates the system has to separate intended rule learning from familiar-template matching, and label corruption is one of the few probes that does it directly.
Few examples work by combining new evidence with a strong prior, so the prior must be tested rather than ignored.
Key idea
Benchmark contamination is especially dangerous here
A model can appear to learn from a handful of examples when the task, benchmark, or closely related items were already present in pretraining. This does not make the model useless, but it changes the interpretation. The observed behaviour may be retrieval or pattern completion from prior exposure rather than rapid learning of a genuinely novel task.
The overlap is measured, not hypothetical. An audit of C4.EN found exact-match contamination running from 1.87% to 24.88% of target texts across four generation test sets, and from under 2% to over 50% of GLUE test inputs. Its authors warned that this is specifically misleading for few-shot and zero-shot claims.
The paper that supplies this lesson's TriviaQA curve says the same thing about itself. Its section on measuring and preventing memorization of benchmarks records: “Unfortunately, a bug resulted in only partial removal of all detected overlaps from the training data.” The model was not retrained. The 64.3 / 68.0 / 71.2 curve remains the best-known illustration of in-context learning, and it was produced by a model whose own authors document a de-contamination filter that did not do its job.
Use private or newly created evaluations when novelty matters. Vary surface forms and test counterfactual tasks that preserve the format while changing the rule.
A few-shot claim is strongest when the task is demonstrably novel to the model.
Steps
A robust few-shot evaluation protocol
The protocol should measure sensitivity, not hide it: define the task family, sample many support sets, randomize order and labels, compare adaptation modes, report distributions rather than a single figure, and test novelty where contamination is plausible. Two of those steps already have documented answers. Both point the same way.
Step four asks you to benchmark against ordinary supervised learning instead of assuming the meta-learner wins. Chen and colleagues ran exactly that comparison across a domain shift, meta-training on mini-ImageNet and testing on CUB: “As shown in Table 3, the Baseline outperforms all meta-learning methods under this scenario.” With a ResNet-18 backbone, the 5-shot numbers are 65.57 ± 0.70% for the simple Baseline against 62.04 ± 0.76% for Baseline++, 62.02 ± 0.70% for ProtoNet, 57.71 ± 0.73% for RelationNet, 53.07 ± 0.74% for MatchingNet and 51.34 ± 0.72% for MAML. The ordinary baseline comes first, and MAML last. Tian and colleagues reached the same conclusion without any domain shift at all: a supervised embedding plus a linear classifier scored 62.02 ± 0.63% (1-shot) and 79.64 ± 0.44% (5-shot) on miniImageNet, and 64.82 ± 0.60% / 82.14 ± 0.43% with self-distillation, above the meta-learning methods they tabulate.
The same step covers retrieval, and retrieval carries the largest single effect in this lesson. With GPT-Neo as both scorer and inference model, randomly sampled prompts scored 1.7 LF-EM on BREAK, 7.3 EM on MTOP and 8.9 EM on SMCalFlow. Plain BM25 retrieval scored 26.0 / 52.9 / 46.1, and a trained retriever, EPR, 31.9 / 64.2 / 54.3. Liu and colleagues found the same direction with GPT-3: nearest-neighbour retrieved demonstrations raised NaturalQuestions accuracy from 28.6 ± 0.3 to 41.6, and WebQuestions from 41.0 ± 0.5 to 50.6. Their abstract concludes that “the retrieval-based prompt selection approach consistently outperforms the random baseline”.
From 1.7 to 31.9 on one benchmark is not a tuning detail. Which examples you put in the prompt dominates how many you put in it. Any protocol that varies the shot count while leaving example selection unspecified is measuring the smaller of the two variables.
1. Define the task family
Specify what varies across episodes and what remains shared.
2. Sample many support sets
Avoid a single curated demonstration set.
3. Randomize order and labels
Test dependence on presentation artifacts.
4. Compare adaptation modes
Benchmark prompting, retrieval, probing, and fine-tuning where relevant.
5. Report distributions
Show medians, spread, worst episodes, and task slices.
6. Test novelty
Use private tasks or counterfactual rules when contamination is plausible.
Example
A demonstration-sensitivity report
A credible few-shot result describes how behaviour changes when the small support set changes. Each line below has a published magnitude attached to it.
- Example identity: Repeat the evaluation with several independently sampled support sets rather than one favourable set, and say which sampler produced them — random selection scored 1.7 LF-EM where a trained retriever scored 31.9 on the same benchmark and model.
- Order sensitivity: Permute demonstrations to detect recency or position effects. On a fixed demonstration set the permutation range has been measured from near state-of-the-art to random guess performance, with a 13% relative improvement available from ordering alone.
- Label wording: Vary class names and instructions. Corrupting labels outright cost only 0–5% absolute across twelve models. Changing the instruction template changed the rankings themselves: a study spanning 20 LLMs, 39 tasks, roughly 6.5M instances and more than 175 instruction paraphrases per task found statistically significant differences in performance on 21 of 25 tasks, with T0pp coming first on a BBH task under one paraphrase and ninth under another.
- Class balance: Test support sets with realistic and with deliberately stressed prevalence, and report the stressed case rather than the average of the two.
- Contamination control: Check whether benchmark examples or templates may have appeared during pretraining. The audit of C4.EN found 1.87% to 24.88% of target texts present verbatim across four generation test sets, so the prior on "my benchmark is clean" should be low.
- Abstention: Include cases where the support set is insufficient and the system should decline, and score declining as a correct answer there rather than as a miss.
Key takeaways
- Few-shot names four different settings — fine-tuning, metric-based inference, meta-learning and in-context conditioning — and a number such as MAML's 48.70 ± 1.84% belongs to exactly one of them.
- Episode composition, not the shot count, moves the score: the same prototypical network scored 68.20 ± 0.66% or 64.24 ± 0.72% on the same 5-shot test depending on how many classes its training episodes contained.
- Demonstrations mainly convey the shape of a task rather than its labels — replacing gold labels with random ones cost 0–5% absolute across twelve models — while whether contradicted labels bind at all depends on model scale.
- Presentation is not a neutral wrapper: order alone can span near state-of-the-art to chance, and formatting alone was worth up to 76 accuracy points on LLaMA-2-13B.
- Contamination is measured rather than hypothetical: 1.87% to 24.88% of target texts found verbatim in C4.EN, and a documented bug that left detected overlaps in GPT-3's training data.
- Ordinary alternatives belong in the comparison as baselines, not footnotes: a plain fine-tuned baseline beat every meta-learner tested across a domain shift, and retrieved demonstrations beat randomly sampled ones from 1.7 to 31.9.