Multi-Task and Hybrid Learning Systems

Different components may learn from different signals and operate at different timescales.

The row that is easiest to draw and hardest to earn is human judgment. Regulators have already written down the test it has to pass. The Article 29 Data Protection Working Party set it out in guidelines on automated individual decision-making and profiling, adopted in 2017 and revised in 2018: “The controller cannot avoid the Article 22 provisions by fabricating human involvement. For example, if someone routinely applies automatically generated profiles to individuals without any actual influence on the result, this would still be a decision based solely on automated processing. To qualify as human involvement, the controller must ensure that any oversight of the decision is meaningful, rather than just a token gesture. It should be carried out by someone who has the authority and competence to change the decision.” The European Data Protection Board quotes that passage verbatim in its 2025 guidelines on the interplay between the DSA and the GDPR. So does the Spanish data protection authority, the AEPD.

Four criteria fall out of the passage in the AEPD's 2024 reading of it: competence, training, independence and practical means, including sufficient time per decision. Read that as a component specification. A reviewer without authority, competence, independence or time is not a component of this system. Under Article 22 GDPR the decision is still solely automated, whatever the diagram says.

Multi-task learning trains one system on several targets, often with shared representations and task-specific outputs. Related tasks can provide useful inductive bias and increase effective supervision.

The tasks can also compete. A shared representation that improves a common task may degrade a rare or safety-critical task, especially when losses and dataset sizes are imbalanced.

That competition has been measured exhaustively. A study presented at ICML in 2020 trained every feasible subset of five vision tasks on the Taskonomy dataset: “5 single-task networks, 10 two-task networks, 10 three-task networks, 5 four-task networks, and a single five-task network”. Each grouping was compared against training the same tasks alone. Some pairings helped. Standley and five co-authors report that “the Normals task helps the performance of every other task with which it is trained”. Some did not, and the damage was one-sided. Training semantic segmentation together with 2D edge detection left segmentation 3.07% better than on its own and edge detection 34.64% worse. The average affinity for that pair was −15.78%.

The sign is not a property of the tasks. That is the result that matters in a design review. Rerun with a higher-capacity encoder, the pairwise relationships “do not seem to correlate with the values in Table 2 (Pearson’s r = 0.08)”. Which tasks belong in one network is an empirical question about your network and your data. It is not a taxonomy question.

The same curve has been measured outside vision, where the competing tasks are languages. Masked language models trained on up to 100 languages, on more than two terabytes of filtered CommonCrawl data, show it. At fixed model capacity, XNLI accuracy rises as languages are added from 7 to 15 and then falls: 71.8% for XLM-7 against 67.7% for XLM-100. Conneau and Khandelwal, who published the work with eight co-authors in 2020, named what happens past the peak: “Beyond this point the curse of multilinguality kicks in and degrades performance across all languages.”

What makes that measurement usable is the fix they price. Widening the model — hidden size 768 to 960 to 1152 — brought XLM-30 back level with XLM-7, and was not enough for XLM-100. Wang and two co-authors at Carnegie Mellon confirmed the phenomenon independently in 2020, and showed it also harms low-resource languages, not only high-resource ones. So the design note is not that task weights require care. Adding a target to a shared network without adding capacity is a decision to spend some of the accuracy of every other target. The exchange rate is measurable before you ship.

Composition can occur inside one model or across explicit modules.

The cascade has been measured at community scale in medicine. Computer-aided detection (CAD) in screening mammography is a two-stage system with a human final stage: the model marks suspicious regions, the radiologist decides. A 2015 study in JAMA Internal Medicine compared 495,818 digital screenings read with CAD against 129,807 read without. The screenings came from 323,973 women and were interpreted by 271 radiologists at 66 Breast Cancer Surveillance Consortium facilities between 2003 and 2009. Sensitivity was 85.3% (95% CI 83.6–86.9) with CAD and 87.3% (95% CI 84.5–89.7) without. Specificity was 91.6% with and 91.4% without. The cancer detection rate was identical at 4.1 per 1,000 women screened. Among the 107 radiologists who read both ways, sensitivity was significantly lower with CAD (OR 0.53, 95% CI 0.29–0.97). Lehman and co-authors put the result in one line: “Screening performance was not improved with computer-aided detection on any metric assessed.”

The same shape had turned up eight years earlier. Fenton and co-authors, in the New England Journal of Medicine of 2007, covered 429,345 mammograms from 222,135 women at 43 facilities. Specificity fell from 90.2% before CAD to 87.2% after (P<0.001) and positive predictive value from 4.1% to 3.2% (P=0.01), with no significant change in cancer detection. Neither stage was broken. The marks were real and the radiologists were qualified. “Early errors propagate” and “stage metrics need coordination” are, in this cascade, a specificity of 87.2% where it had been 90.2% and a positive predictive value of 3.2% where it had been 4.1%. That cost exists only in the composition. It appears in no single component's evaluation.

One model cannot responsibly carry every responsibility in a global moderation system.

Since 25 September 2023 a workflow like this has been on the record in Europe. The Digital Services Act obliges providers of online platforms to file a statement of reasons for every content moderation decision. The statements go into the EU's DSA Transparency Database. Its schema forces the split these bullets describe. One field records “whether automated tools were used to detect non-compliant content”. A separate field records whether the decision itself was made in a “fully automated, partially automated, or in non automated manner”.

One ten-day window of that database has been analysed in detail. It held more than 131 million statements, filed between 19 and 28 November 2023. That is about 12 million a day. Kaushal and four co-authors found practices that diverge completely: “X, LinkedIn and Booking.com do not automate at all, Facebook, Instagram, Snapchat, and YouTube make automated decisions for some SoRs, whereas AliExpress, Google Maps, Google Shopping, and TikTok make fully automated decisions for most (or all) SoRs.”

The law also fixes where the human sits. Article 20 of the Digital Services Act gives recipients at least six months to lodge a complaint against removal, account suspension or demonetisation. It requires the provider to reverse the decision without undue delay when the complaint has sufficient grounds. Article 20(6) then closes the loop: “Providers of online platforms shall ensure that the decisions, referred to in paragraph 5, are taken under the supervision of appropriately qualified staff, and not solely on the basis of automated means.” The European Data Protection Board restates the obligation in its 2025 guidelines. First-line detection may be automated; the decision on the appeal may not. The last two bullets below are not a design preference. They are an interface the regulation specifies.

What the seams of such a stack actually cost is also on the record. On 15 May 2025 the Meta Oversight Board decided fifteen cases together, all of them Facebook and Instagram posts about breast cancer removed under the Adult Nudity and Sexual Activity policy. Meta reinstated all fifteen only after the Board raised them. The Meta Transparency Center's own account of the bundle says they were “removed … in error”.

The Board was checking whether an old recommendation had worked. It had made that recommendation on 28 January 2021, deciding a case where the post “was detected and removed by a machine learning classifier trained to identify nudity in photos”. Meta's figure for the repaired pipeline — text-overlay improvements plus a new health content classifier, in place since July 2021 — describes a routing change rather than a better classifier: “Meta reported that its implementation has contributed to an additional 1,000 pieces of content being sent for human review that would have previously been removed between March 21 and April 18, 2023, alone.” The fix was a thousand items in four weeks, moved from the removal branch to the human branch.

The human branch is the one that runs out. Meta declined the companion recommendation to guarantee human review of appeals against automated nudity removals, saying appeals go to a content reviewer “except in cases where [the company has] capacity constraints”. A hybrid workflow degrades where its scarcest component is. That is almost never the model.

The system specification should expose interfaces and failure propagation.

Step 6 is the step teams treat as a formality. There is a preregistered meta-analysis of what it finds when they do not. Published in Nature Human Behaviour on 28 October 2024, it pooled 370 effect sizes from 106 experiments published between January 2020 and June 2023. Its headline finding, from Vaccaro and two colleagues at the MIT Center for Collective Intelligence: “First, we found that, on average, human–AI combinations performed significantly worse than the best of humans or AI alone (Hedges’ g = −0.23; 95% confidence interval, −0.39 to −0.07).”

Read the second number alongside the first. The same pooled data has those combinations performing better than humans alone (g = 0.64, 95% CI 0.53 to 0.74). A component review would have passed: the humans improved. The system still lost to its own best part. The losses concentrated in decision tasks (g = −0.27) while the gains were in content creation (g = 0.19). That is a statement about where in your product a hybrid design is likely to subtract.

A separate experiment supplies a mechanism. Bansal and seven co-authors, at CHI in 2021, found that “explanations increased the chance that humans will accept the AI’s recommendation, regardless of its correctness”. The interface intended to make the human stage more critical made it more compliant. Composition is not additive, and no dashboard of component metrics can tell you it failed.

A rule may depend on a model score, a retriever may depend on learned embeddings, and reviewer labels may depend on model suggestions. These dependencies can create feedback loops and coordinated failure.

Document data and control flow. Modular diagrams are useful only when they reveal real coupling rather than drawing clean boxes around tangled behavior.

The loop this paragraph warns about has been simulated. A 2018 RecSys paper builds a synthetic world in which users' true preferences are known. The recommender then trains on the ratings its own earlier recommendations produced. That is the coupling of “reviewer labels may depend on model suggestions”, run forward in time. Chaney, Stewart and Engelhardt state the finding plainly: “using data confounded in this way homogenizes user behavior without increasing utility.” The system's inputs are downstream of its own outputs. A metric computed on them measures the loop rather than the world. That failure appears in no component's contract. It is visible only in a diagram that draws the arrow back.

The boxes fail to hold for accountability as well, and a court has said where the line actually falls. Case C‑634/21, OQ v Land Hessen — the SCHUFA case, decided on 7 December 2023 — put a two-component arrangement in front of the Court of Justice of the European Union. A credit information agency computes a probability value about a person's ability to meet future payment commitments. A bank decides the loan. One score crosses one interface. The referring court, the Verwaltungsgericht Wiesbaden, had found that “an insufficient probability value leads, in almost all cases, to the refusal of that bank to grant the loan applied for”.

Because the third party receiving the value “draws strongly” on it, the Court held that establishing the value is itself “automated individual decision-making” within Article 22(1) GDPR. The European Data Protection Board relies on the same judgment in its 2025 guidelines, for the point that Article 22(1) “lays down a prohibition in principle”. The scoring team's contract said it emitted a number and made no decision. The Court read the arrow instead of the box.

Each learned or deterministic stage should publish a small operational contract. The human stages need one as much as the models do. The AEPD's four criteria — competence, training, independence and practical means, including sufficient time per decision — are the fields a review stage has to fill in before anyone may call it human oversight.