Multi-Task Learning and Mixture-of-Experts

Multi-task learning trains several objectives together. Related tasks can then share representations, examples, or regularization, and a low-data task may benefit from features learned on a high-volume relative. Auxiliary tasks can force the encoder to preserve information that improves generalization on the primary task.

The same sharing can cause negative transfer. Gradients from one task may move shared parameters in a direction that harms another. Differences in label noise, scale, sampling frequency, or loss magnitude let one task dominate. A task that is semantically related but measured poorly can be worse than an unrelated clean one.

Mixture-of-experts introduces conditional computation. A router selects a subset of expert subnetworks for each input or token, so parameter capacity grows without activating every expert every time. It does not remove multi-task tradeoffs. It moves some of them into routing, specialization, communication, and capacity allocation.

The claim is old, and it was measured. Multitask learning “improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias”, Rich Caruana wrote in 1997. On the simulated 1D-ALVINN steering task his shared nets beat single-task nets: “on the important steering task, MTL outperforms STL 15–30%”. Both were trained on “exactly the same training patterns”. Only the extra output signals differed.

What that argument does not supply is a way to know in advance which tasks will do this for each other. The paper that asked the question asked it plainly: “Unfortunately, this often leads to inferior overall performance as task objectives can compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning?” Standley and colleagues then answered it the expensive way. In 2020 they trained every one of the 31 subsets of a five-task set on the Taskonomy dataset — about 4 million examples from 3D scans of about 600 buildings, with no building overlap between train and test. Some groupings paid. Semantic segmentation trained alongside depth came out 4.17% better than trained alone, on a half-size network. Some pairs hurt both tasks. And their measured pairwise affinities did not track Taskonomy's own transfer affinities: Pearson's r = −0.14. A published taxonomy of task relatedness did not predict which tasks belonged in one network. Relatedness is a property of your setup. It is measured there or not at all.

A router converts each input representation into expert assignments and weights. Mixtral 8x7B makes the arity concrete. Each layer is composed of 8 feedforward blocks, and a router network selects two of them for each token, at every layer. Mistral's own paper states the consequence: “As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference.”

Those are two numbers, and they are not interchangeable. 47B is what has to be resident. 13B is what does the work per token. Hugging Face's write-up of the same model pins both halves down from outside the lab that built it. The arithmetic the name suggests is wrong — “The total number of parameters is not 56B, but about 45B.” — and it is the active count, not the total, that sets the speed. Selecting two experts for each timestep “allows the model to decode at the speed of a 12B parameter-dense model, despite containing 4x the number of effective parameters”.

Note also that the two organisations do not agree on the total: 47B against about 45B. That is exactly the kind of number to quote with its source rather than round into a headline. A serving plan that budgets memory from the compute figure, or latency from the memory figure, is planning for a model that does not exist.

The architecture should reflect how much common structure the tasks truly have. Hard sharing bets that a single trunk can serve every objective. Soft sharing buys specialization with parameters and tuning. Sparse routing buys it with conditional computation: in Mixtral 8x7B, 8 feedforward blocks per layer, of which the router picks two per token. The capacity sits in memory whether or not a given token uses it.

None of the three escapes the others' failures. Standley and colleagues found task pairs that hurt both members, and routed models add the load-balancing machinery this lesson takes up under expert collapse.

Conflict is not a single phenomenon, so one weighting trick rarely solves every case. The first item on this list has a test rather than a description: “We define two gradients to be conflicting if they point away from one another, i.e., have a negative cosine similarity.” That is Yu and Finn, in the 2020 gradient-surgery paper. Its §2.2 is titled “The Tragic Triad: Conflicting Gradients, Dominating Gradients, High Curvature” — the first, second and last bullet below, under other names.

The cost of leaving the triad alone is measured. On the Meta-World benchmarks, projecting conflicting gradients (PCGrad) with SAC “successfully solves all of the 10 tasks in MT10 and about 70% of the 50 tasks in MT50”, while a single SAC policy and a multi-head policy are “unable to acquire half of the skills in both MT10 and MT50”.

An aggregate gain can hide a task or an expert that regressed badly. The baselines at the top of this plan are not a formality, and there is a published result to prove it. In 2022 a separate group re-ran the specialised multi-task optimisers — PCGrad among them — against the plainest baseline available, simply summing the task losses. Kurin and Kumar report the outcome: “We show that unitary scalarization, coupled with standard regularization and stabilization techniques from single-task learning, matches or improves upon the performance of complex multi-task optimizers in popular supervised and reinforcement learning settings.” Much of the machinery's advantage, they argue, is regularisation in disguise. They close with a demand: “We believe our results call for a critical reevaluation of recent research in the area.”

That is the reason step 2 comes before step 5. If the summed-loss baseline is never tuned as carefully as the clever method, a gain credited to gradient surgery or to sparse routing may be a gain from regularisation that nobody tried on its own. It will be credited to the wrong component for as long as the comparison is missing.

A shared model should be evaluated as a network of task interactions rather than one aggregate score. Every layer of that audit has a measurable quantity attached to it, and this lesson has named one for each. Whether shared representations pay: 4.17% for semantic segmentation trained alongside depth, and pairs that hurt both tasks. Whether task gradients conflict: negative cosine similarity, logged per step. What routing costs on both axes: 22 TPU v3 core-years against 235.5, and 47B parameters resident against 13B active per token. And what the system does when a task label, an expert or a route is unavailable. A single headline score can be produced while every one of those is unmeasured.