Multi-Task and Multi-Objective Optimization

A common encoder can learn features that help several tasks, reduce memory, and pool evidence. It can also receive gradients that demand incompatible changes.

Multi-task optimization therefore asks two questions: how hard each task gets to pull, and whether their directions cooperate in shared parameters.

The simplest answer to the first question is to add the losses with equal weight. It has a measured price. Sener and Koltun ran it in 2018 across CelebA's 40 attribute tasks. Mean per-task error was 9.62% under uniform scaling, against 8.77% for separate single-task models. On Cityscapes, uniform scaling reached 54.59 mIoU for segmentation where the single-task model reached 60.68 mIoU. Their own verdict: “Although uniform scaling is the norm in the multi-label classification literature, single-task performance is significantly better.”

That is not one team's dataset quirk. Vandenhende and colleagues built their own benchmark for IEEE TPAMI and scored uniform weighting at -3.99% average relative improvement over carefully trained single-task baselines on PASCAL, with MGDA at -6.81%. Two groups, four datasets, the same sign. The scalar losses were equal. The learning was not.

The interaction depends on data, objective, and parameter sharing. Task sampling decides which tasks and labeled examples appear in each batch. Loss reduction decides how each task normalizes and weights its valid targets. Task gradients carry a magnitude and a direction into the shared parameters. The shared representation then changes for every head at once. And the outcomes come apart: validation may improve for one task while another deteriorates.

The first layer decides more than it looks. The XLM-R team at Facebook AI held capacity fixed and changed only how many languages shared it. “Specifically, the overall XNLI accuracy decreases from 71.8% to 67.7% as we go from XLM-7 to XLM-100,” they reported in 2020. They named the effect in the same paper: “We refer to this tradeoff as the curse of multilinguality”.

Nothing in that experiment changed a loss coefficient. What changed was how often each task appeared, and how much shared representation each had to itself.

A task with sparse labels may update its head and shared encoder far less often than another task. Zero loss on unlabeled batches should not be mistaken for evidence that the task is solved.

Each method encodes assumptions and adds state.

Each of those names belongs to a paper with a claim you can check. Uncertainty weighting arrived at CVPR in 2018, from Kendall and colleagues. Their loss weighs tasks by “the homoscedastic uncertainty of each task”. A single model learned depth regression, semantic segmentation and instance segmentation together, and it beat “separate models trained individually on each task”.

Their own Table 1 shows how much the weighting choice moves the result. On Tiny CityScapes, at 128x256 and 50,000 iterations, segmentation reached 59.4% IoU trained alone. An unweighted sum of the three losses dropped it to 50.1% IoU. Approximately optimal fixed weights of 0.89/0.01/0.1 lifted it to 62.8%. Three-task uncertainty weighting reached 63.4%, with learned weights ending training in the ratio 43 : 1 : 0.16. “Using a uniform weighting results in poor performance, in some cases not even improving on the results from the single task model,” the authors write. The model was the same in every row. Only the weighting changed.

GradNorm followed at ICML the same year. It tunes gradient magnitudes and exposes only “a single asymmetry hyperparameter”. That is the whole tuning surface, whatever the number of tasks. The alternative was priced in the same paper: “Even after 100 networks trained, grid search still falls short of our GradNorm network,” write Chen and colleagues.

The reason this lesson still says cautious is that independent re-runs disagree about the sign. The TPAMI benchmark records that “GradNorm is the only technique that outperforms training a separate model for each task” — on NYUD-v2. On PASCAL the same method scores -3.97% in that same paper. And Fifty and colleagues reported in 2021 that GradNorm can sometimes perform worse than training all tasks together. One method, three verdicts.

Cosine similarity depends on batch noise, parameter subset, scaling, and current model state. A negative value on one step does not prove harmful transfer.

Track distributions over time and connect them to task outcomes. Methods that manipulate gradients can also change convergence or favor one task implicitly.

PCGrad is precise about where it acts. It projects one task’s gradient “onto the normal plane of the gradient of any other task”, and it does so only where two gradients actually conflict. Yu and colleagues gave it that definition at NeurIPS 2020, and the same paper identifies three landscape conditions that cause “detrimental gradient interference”. A cosine reading is where that diagnosis starts, not where it ends.

Compare sharing against simpler alternatives. Build single-task baselines with their own training budget. Add a shared-feature baseline with simple fixed task weights. Log task-level valid counts, losses, norms, cosine distributions and update cadence. Test one balancing method under matched compute. Evaluate every task, including worst-task regressions and calibration. Then price the architecture against memory, latency, maintenance and missing-task behavior.

Step four is where two NeurIPS 2022 evaluations set the bar. Kurin and colleagues at Oxford and Berkeley benchmarked IMTL, MGDA, GradDrop, PCGrad and RLW on Multi-MNIST, 40-task CelebA, Cityscapes and Metaworld MT10/MT50. Their abstract states the result: “We show that unitary scalarization, coupled with standard regularization and stabilization techniques from single-task learning, matches or improves upon the performance of complex multi-task optimizers in popular supervised and reinforcement learning settings.” Xin and colleagues at Google Research ran WMT translation pairs of up to 40,853,298 training examples, plus CelebA, against MGDA, GradNorm, PCGrad, IMTL and RLW. Their conclusion: “all of the MTO algorithms in our study simply yield performance trade-off points on the scalarization Pareto front. As such, their performance can be fully replicated by simply optimizing a weighted average of the losses”.

Step six is what those methods cost while doing it. “In fact, Figure 4(d) shows that a single unitary scalarization run takes roughly 15 hours, whereas PCGrad, MGDA and GradDrop require more than a week,” the Oxford and Berkeley team writes. The Google group measured the same trade in throughput. A 120M-parameter translation model ran at 11.52 steps/sec under scalarization and 4.81 steps/sec under MGDA. A 187M-parameter 12-layer model trained by plain scalarization still ran at 5.48 steps/sec. The extra capacity was cheaper than the extra optimizer. Search over task groupings is priced too: Fifty and colleagues put the exhaustive alternative at “an additional 2,008 TeslaV100 GPU hours to find task groupings”. Compute spent balancing gradients is compute not spent on a bigger model, more data, or the baseline you still owe each task.

These problems can coexist, and they need different diagnostics. Magnitude imbalance comes from reduction or scale. It shows up as norm disparity, is answered with weights or normalization, and risks hiding true difficulty. Directional conflict comes from competing representations. It shows up as negative cosine, is answered with architecture or gradient surgery, and rests on noisy estimates. Missing supervision comes from annotation availability. It shows up as empty loss components, is answered with masking and sampling, and produces uneven update cadence.

The verdict also depends on the baseline you chose before measuring anything. In that exhaustive ICML 2020 sweep, the five-task Taskonomy network was 19.00% worse in total loss than five equal-sized single-task networks. It was 0.34% better than five networks each given a fifth of the compute. Same model, two defensible comparisons, opposite conclusions. “In Setting 1, multi-task networks do not compare favorably to multiple single-task networks that are each allowed the same computational budget as the single multi-task network,” write Standley and his co-authors. A year later, Fifty and colleagues repeated the same five Taskonomy objectives and beat training-everything-together by 10.0%, simply by choosing which tasks to group.

Track optimizer steps containing valid supervision per task. Sampling and accumulation may need to guarantee minimum exposure without distorting prevalence silently.

The largest published demonstration of that is a translation model. Google trained a single model over 103 languages and over 25 billion examples. Equal sampling maximises transfer to the low-resource languages, but “this also has the side-effect of significantly deteriorated performance on high resource languages”. No coefficient was tuned to produce that. The sampling rate did it. Who appears in the batch, and how often, is the weighting that actually ran.