Curriculum Learning, Hard Examples, and Data Scheduling

Standard stochastic training treats example order as random noise. Curriculum learning deliberately changes that order or sampling probability. A model might begin with clean, short, or prototypical examples before receiving ambiguous cases. Hard-example mining does the opposite later in training: it concentrates effort on examples the current model finds difficult. These methods alter the optimization path and the effective training distribution. They can speed convergence or improve rare-case learning. But they can also create blind spots. “Easy” often means easy for the current model or well represented in the majority data, not inherently simple. “Hard” may mean mislabeled, out of support, or affected by a broken feature pipeline. The schedule is therefore a model component. It needs ablations, fairness checks, and controls against repeatedly sampling noise.

The founding experiment was about shapes. A network trained first on simple shapes, then switched to the harder set, over “20 different random seeds”. Training was capped at 256 epochs either way. The switch epoch was varied over “0 (the baseline with no curriculum) and the powers of 2 until 128”. Bengio and colleagues reported in 2009 that “the best generalization is obtained by doing a 2-stage curriculum where the first half of the total allowed training time (of 256 epochs) is spent on the easier examples rather than on the target examples”. Half the budget spent on the easy half of the data. Everything after it in this lesson is the bill for that result.

Each strategy uses a different source of difficulty information, and two of the three have been priced in a published experiment.

A hand-designed curriculum orders examples using domain knowledge or known complexity — short sequences before long ones. It is easy to explain and can encode expert structure, but the ranking is somebody's opinion and it cannot adapt once training starts.

Self-paced learning hands the ranking to the model. Loss or confidence decides which examples are admitted, and the admitted set expands from the easier examples outward. Kumar and colleagues introduced it in 2010 and measured what it buys on a motif-finding task: “Indeed, over all the 100 runs (5 proteins, 5 folds and 4 seed values) CCCP got stuck in a bad local minimum 18 times (where a bad local minimum is one that gave 50% test error) compared to 1 run where self-paced learning got stuck.” Eighteen catastrophic runs in a hundred became one. The price is in the same table: an average of 1,287 seconds per run against 824 for CCCP, roughly 56% more wall-clock time. Adapting to the current learner is real, it is measurable, and it is not free. And because the learner's own early mistakes decide what it sees next, it still needs safeguards for the cases it chooses to ignore.

Hard-example mining oversamples high-loss, confusing, or near-boundary examples. It is standard practice for detection and retrieval negatives. But hard selection is itself a cost, because discarding easy examples is not the same as de-emphasising them. The focal loss merely down-weights easy examples instead of dropping them, and on COCO in 2017 it beat mining outright with a ResNet-101-FPN RetinaNet: “These results use ResNet-101, our baseline trained with FL achieves 36.0 AP for this setting. In contrast, the best setting for OHEM (no 1:3 ratio, batch size 128, nms of .5) achieves 32.8 AP. This is a gap of 3.2 AP, showing FL is more effective than OHEM for training dense detectors.” The mining policy lost 3.2 AP to a policy that kept every example and reweighted it. Caps and freshness controls are the minimum. The deeper question is whether the schedule needs to select at all.

No single difficulty score is universally safe, and two measured comparisons show that the choice of proxy decides the outcome.

Score WinoGrande examples by how much a model's confidence in them varies across epochs, then train RoBERTa-large on a third of the data. Which third you keep changes everything. The most ambiguous third scored 78.7 in-distribution and 87.6 out-of-distribution on WSC, above the full training set's 86.0. The hard-to-learn third scored 77.9 and 87.2. The highest-confidence third — the data an easy-first curriculum would start with — managed only 69.4 and 83.9, worse than a random 33% at 73.3 and 85.6. Swayamdipta and colleagues summarised it in 2020: “Training on the most ambiguous data results in the best OOD performance, exceeding that of 100% train, even with just a third of the data.” Confidence and the variability of confidence are not two versions of the same signal. They rank the dataset almost oppositely.

Crude structural proxies are not worthless either. Order training sentences by source-sentence length and by word rarity, admit them as the model's competence grows, and both quality and training time improve. On WMT-16 En→De a Transformer went from 27.95 to 30.16 BLEU at 0.55 of the baseline's training time, and En→Vi reached the baseline in as little as 0.31 of it. Platanios and colleagues put it this way in 2019: “We show that our framework can help improve the training time and the performance of both recurrent neural network models and Transformers, achieving up to a 70% decrease in training time, while at the same time obtaining accuracy improvements of up to 2.2 BLEU.” A proxy that knows nothing about meaning still bought both accuracy and time.

When the model controls sampling, prediction errors influence the data it will see next. The loop runs in five stages. Train on the current sample. Score difficulty by loss, uncertainty, margin, or retrieval hardness. Update the sampling weights according to the policy. Retrain and rescore, at which point the changed model produces a new notion of difficulty. Then audit the loop for coverage, noise concentration, and subgroup exposure. Only the last stage is optional in practice, which is why it is the one that gets dropped.

The control condition should use the same examples with ordinary random sampling. The reason is that, run at scale, the control tends to win. Wu and colleagues ran thousands of example orderings in 2021, across curriculum, anti-curriculum and random-curriculum training. Their finding: “We find that for standard benchmark datasets, curricula have only marginal benefits, and that randomly ordered samples perform as well or better than curricula and anti-curricula, suggesting that any benefit is entirely due to the dynamic training set size.” The same abstract then names the conditions under which the ordering does matter: “curriculum, but not anti-curriculum or random ordering can indeed improve the performance either with limited training time budget or in the existence of noisy data”.

That is the hypothesis to build the experiment around — a limited budget, or noisy labels — rather than a general improvement. Six steps make it testable. Define the baseline sampler and record uniform or class-balanced exposure. Specify the difficulty score, stating what it measures and which artifacts may inflate it. Bound the sampling weights so that no valid group disappears and no example dominates. Audit labels among the hard cases and estimate the fraction of annotation errors or ambiguous targets; in RocketQA's audit that fraction was about 70%, so the estimate is not a formality. Compare full learning curves for convergence speed and final generalization, since a schedule that only arrives earlier is a budget result, not an accuracy result. Then retest calibration and slices, because scheduling changes the effective class and group distribution.

An example that was difficult in the first epoch may become routine later, while another can remain hard because its label is wrong. Static rankings therefore mix genuine learning progress with stale information.

One way to separate the two is to count, for each training example, how often a network that had already learned it forgot it again. Toneva and colleagues did exactly that in 2019. Across 5 seeds, 15,628 CIFAR-10 training examples — 31.3% of the set — were never forgotten at all; the same statistic reaches 55,012 and 45,181 examples, 91.7% and 75.3%, on the other datasets they measured. Those stable examples are not merely easy. They are removable: 30% of CIFAR-10 could be deleted by forgetting score with comparable generalization, and 35% with under 0.2% degradation. Then the authors randomised 20% of the labels and looked at the other end of the ranking: “We observe that the most forgotten examples are those with noisy labels and that no noisy examples are unforgettable.” An independent group restated the finding two years later — “Toneva et al. demonstrate that examples with low forgetting score may be completely omitted during training without any noticeable effect on the accuracy of the learned predictor” — and pruned 50% of CIFAR-10 without affecting accuracy.

So the two ends of a difficulty ranking mean different things, and only one of them is worth oversampling. Refresh difficulty on a controlled schedule. Preserve an exploration fraction that samples outside the current hard set. Otherwise the model may train on a narrowing view of the data and stop discovering new failure modes.

A curriculum changes the effective training distribution, so somebody has to version it and measure what it did. Define difficulty explicitly, stating whether it comes from loss, margin, rarity, sequence length, or domain knowledge. Separate noise from challenge by reviewing high-loss examples for label errors and ambiguous targets; this is the step that stands between 26.03 and 36.38 MRR@10 on the same set of mined negatives. Preserve coverage with minimum exposure for rare classes, domains, and long-tail conditions. Report the worst group beside the average, because the gap between 47.2% and 81.1% on CelebA is invisible in an average that moved by less than eight points. Refresh scores rather than reusing difficulty estimates from an earlier model state, since 31.3% of CIFAR-10 was never forgotten at all and the most-forgotten examples were the corrupted ones. Compare fixed and adaptive schedules against ordinary sampling. Across thousands of orderings, random ordering performed as well as or better than curricula and anti-curricula on standard benchmarks. Then recheck calibration, because sampling changes alter score distributions and threshold behavior.