Curriculum Learning as Transport: Understanding Curricula with Wasserstein Geodesics

AI summary

Overview

Research area: Machine learning — curriculum learning, dataset ordering, and optimal transport (Wasserstein geometry).

Technical level: Intermediate. The core idea is intuitive (a curriculum as a path through difficulty levels), but the optimal-transport formulation and the experimental design require comfort with probability distributions and training dynamics.

Scope: The paper introduces a unified optimal-transport framework for representing curricula as Wasserstein geodesic paths over discrete difficulty levels, then uses a calibrated synthetic benchmark suite to isolate which curriculum design choices (ordering, exposure, endpoint smoothness, pacing, geometry) actually drive observed gains.

What This Paper Is About

Curriculum learning — training a model on examples ordered from easy to hard — is widely used but poorly understood, because most methods change several design choices simultaneously: how difficulty is defined, the direction of ordering, how much exposure each difficulty level receives, and how fast training moves between levels. When a curriculum helps, it is usually unclear whether the gains come from ordering itself, from extra exposure to easy examples, or from smoother transitions. This paper recasts a curriculum as a trajectory of sampling distributions over difficulty levels and uses Wasserstein (optimal transport) interpolation to build that trajectory, which lets the authors vary one design choice at a time while holding the others fixed.

Key Contributions

1. A transport-based formalism for curricula. The paper represents a curriculum as a path {P_t} between an easy-heavy start distribution P_0 and a hard-heavy end distribution P_1 over L ordered difficulty levels, together with a schedule t_k mapping training step to path position. Wasserstein geodesic interpolation moves probability mass smoothly through adjacent levels, unlike linear interpolation, which directly mixes the endpoints.

2. A controlled synthetic benchmark suite. The authors construct and calibrate 12 tasks with 33 task-by-difficulty-axis conditions (e.g., k-Parity, Dyck language, Dyck error detection, Survo puzzles, calendar scheduling, Game of 24), each varying exactly one difficulty attribute while holding task semantics fixed, with per-task budgets calibrated from pilot static-i.i.d. learning curves.

3. Empirical decomposition of curriculum design factors. The framework isolates ordering direction, exposure matching, endpoint sharpness, traversal speed, and level geometry within a single parameterization, producing the paper's main empirical findings about which factors matter and when.

4. Two extensions beyond the basic path. A prototype adaptive-geometry method called warp (Wasserstein-Adaptive Reshaping of Paths), which learns level distances from gradient alignment during a short warmup, and an extension of the framework to structured difficulty graphs (a cube and a tree built from arithmetic operators) rather than one-dimensional orderings.

Main Findings

• No universal winner. Across the 33 conditions and three budget regimes, static i.i.d., linear, and Wasserstein curricula each win substantial subsets of conditions, and the ranking shifts with budget. Curriculum effectiveness depends jointly on task, difficulty axis, and available budget.

• Curricula reallocate a fixed budget rather than uniformly improving it. Using exposure-adjusted accuracy (final accuracy divided by cumulative exposure to a difficulty bucket), static i.i.d. is strongest on the easiest bucket, linear is strongest on the middle bucket, and Wasserstein is strongest on the hardest bucket. On the hard bucket, Wasserstein beats linear on exposure-adjusted accuracy in all 99 task-by-axis-and-budget conditions.

• Ordering matters beyond cumulative exposure. An exposure-matched static baseline — which preserves total exposure to each level but removes the temporal progression — does not recover the hard-bucket gains of easy-to-hard Wasserstein. Hard-to-easy Wasserstein frequently harms performance outright: forward Wasserstein beats reverse on the hardest level in 92 of 99 conditions.

• Pacing and endpoint sharpness matter within easy-to-hard. Sweeping a 9×9 grid over endpoint temperature τ and pacing exponent γ on 20 conditions reveals a broad good region with moderate pacing and smoother endpoints, rather than a single optimum. Very sharp endpoints are brittle under both overly aggressive and overly conservative pacing.

• Adaptive geometry can improve pacing. The warp prototype, which learns level distances from gradient similarity, beats fixed Wasserstein at small (+0.2 points) and medium (+0.9 points) budgets and matches it at large budget — a proof-of-concept that learning geometry is a viable route to learned pacing.

• The framework extends to structured difficulty spaces. On a cube and a tree of arithmetic-operator tasks, geometry-aware curricula help most near the hard end; Wasserstein consistently improves the hardest cube node, while warp dominates in the small-budget tree setting and Wasserstein leads overall at large budget on the tree.

• Effects weaken on real data. The authors report (Appendix D) that in supervised fine-tuning settings with pretrained models and noisier difficulty labels, curriculum effects are much weaker and harder to disentangle.

Methodology in Plain English

The researchers start by partitioning each task's training data into ordered difficulty levels. A curriculum is then just a sequence of probability vectors — at each training step, a batch is drawn from level ℓ with probability P_ℓ. Instead of hand-designing schedules, they interpolate between two endpoint distributions using Wasserstein geodesics, which move probability mass through neighboring levels along the shortest "transport" path, rather than jumping between mixtures as linear interpolation does.

This separation is the key move. The endpoints determine where training starts and ends; the geometry determines which levels count as "nearby"; the schedule determines how fast training traverses the path. Because these are independent knobs, the authors can hold everything fixed and change only the ordering direction (forward vs. reverse), or only the endpoint sharpness (via a temperature parameter τ), or only the pacing exponent (γ), and attribute any performance difference to that single factor.

They validate this on a synthetic suite designed so that each task has one clean difficulty axis. Budgets (small, medium, large) are calibrated per task from pilot baseline runs, so that "small" means an early stage of learning on every task rather than an early stage on some and a late stage on others — a detail that makes cross-task comparison meaningful. They then extend the same construction to graph-structured difficulty spaces, where levels are nodes connected by edges rather than arranged in a line.

Why This Matters

Impact on research. The paper reframes a crowded, heuristic-driven subfield around a small set of separable variables. By showing that ordering helps beyond cumulative exposure, and that pacing and endpoint smoothness can flip a curriculum from helpful to harmful, it gives a concrete recipe for controlled curriculum experiments and a language for reporting what a curriculum method actually changed.

Real-world applications:

• Training data scheduling for language models, where deciding how much of the budget to spend on easy versus hard examples directly affects final capability on the hardest cases.
• Efficient fine-tuning under tight compute, where a curriculum that improves hard-bucket accuracy per unit of exposure is valuable when hard examples are scarce or expensive to generate.
• Robotics and reinforcement learning, where tasks are naturally organized along difficulty and prior OT-based curriculum work already exists; this framework offers a way to separate pacing effects from task-ordering effects.
• Data curation and active selection pipelines, where the "exposure-adjusted accuracy" diagnostic can reveal that a curriculum is simply spending budget on levels that no longer need it.

Industry relevance. Practitioners routinely choose data mixtures and ordering heuristics without a way to attribute gains. The distinction between "more exposure to easy data" and "better timing of when hard data appears" has direct implications for budget allocation, and the warp prototype suggests that learned, data-driven pacing is a tractable improvement over fixed schedules.

Future Directions

• Learning geometry or pacing from transfer signals. The warp prototype is deliberately minimal; learning geometry from richer diagnostics — rather than gradient cosine similarity alone — is a natural next step.
• Transfer to pretrained and natural-data settings. The authors explicitly flag that their effects weaken substantially in SFT with pretrained models and noisy difficulty labels, leaving open how much of the framework survives outside synthetic, from-scratch training.
• Richer difficulty structures. The cube and tree experiments are illustrative; defining Wasserstein curricula over more complex graphs, or difficulty spaces that interact with prior model knowledge, remains unexplored.
• Disentangling the hard-bucket advantage. The paper's bridge-effect and level-to-level transfer experiments hint at why forward curricula help hardest levels, but a mechanistic account is not yet established.

Target Audience

This paper suits machine learning researchers working on curriculum learning, data mixing, and training data scheduling, as well as theoretically inclined readers interested in optimal transport applied to training dynamics. Practitioners designing data mixtures or ordering pipelines for large-scale training will find the exposure-adjusted accuracy diagnostic and the ordering-versus-exposure distinction immediately useful, though they should note the authors' own finding that effects attenuate in pretrained, natural-data settings. Readers without prior exposure to optimal transport will need to engage with the Wasserstein interpolation definition in Section 3.1, but the surrounding experimental logic is accessible.

Authors’ abstract

Curriculum learning is governed by several coupled design choices---how difficulty is defined, how examples are ordered, how much exposure each level receives, and how quickly training moves across levels---making it hard to isolate what actually helps. We present Wasserstein curriculum paths, a simple transport-based framework that decouples these factors by representing curricula as trajectories of training distributions over discrete difficulty levels. Across a calibrated synthetic suite with 12 tasks and 33 difficulty axes, we use this framework to isolate the effects of ordering, matched exposure, endpoint smoothness, and pacing under fixed training budgets. We find that curriculum effects are strongly context-dependent: no single strategy dominates across tasks, difficulty axes, and budgets, and curricula mainly change where a fixed budget is spent most effectively. Within this framework, easy-to-hard ordering improves hard-level performance relative to exposure-matched static sampling, showing that the benefit is not explained by cumulative exposure alone. We further show that endpoint smoothness and pacing substantially affect where along the difficulty spectrum a curriculum is effective. Finally, we show that the same transport view naturally supports extensions to learned pacing through geometry and to structured difficulty spaces beyond one-dimensional orderings.