Skip to content
AI.info

Advanced techniques

Advanced Techniques: A Decision Map

Build a practical map of advanced machine learning techniques and learn when their extra complexity is justified.

By the end you can

Why advanced does not mean automatically better

Advanced machine learning is not a shelf of upgrades. It is a set of interventions for cases where a well-built standard workflow has run into a limit somebody has already named. An ensemble may reduce variance. Transfer learning may compensate for scarce labels. Self-supervision may extract signal from unlabeled data. Offline reinforcement learning may be the only realistic option when live exploration is unsafe.

The mistake is to select the method first and invent the problem later. A disciplined team starts with evidence. Which examples fail? Is the failure caused by data scarcity, distribution shift, correlated model errors, delayed rewards, catastrophic forgetting, or relational structure? Only then does it choose the smallest method whose mechanism matches that failure.

The measured record is blunter than the enthusiasm. In 2023 Duncan McElfresh and his co-authors ran the comparison at scale — 19 algorithms, 176 datasets — and their abstract reports: “we conduct the largest tabular data analysis to date, comparing 19 algorithms across 176 datasets, and we find that the 'NN vs. GBDT' debate is overemphasized: for a surprisingly high number of datasets, either the performance difference between GBDTs and NNs is negligible, or light hyperparameter tuning on a GBDT is more important than choosing between NNs and GBDTs.” Nineteen algorithms, one hundred and seventy-six datasets. On a surprisingly high number of them, the decision that moved the score was a tuning run on the simple learner. So the question this path asks is not which family wins. It is what has to be true of your failure before the more complicated family is worth owning.

This path treats advanced techniques as design choices with costs. Every gain introduces new assumptions, validation requirements, operational risks, or debugging difficulty. The goal is not to collect algorithms. It is to decide when the extra machinery earns its place.

An advanced method is justified by a diagnosed bottleneck and a measurable improvement, not by novelty.

Case

Forty-five tabular datasets, and the trees stayed ahead

Forty-five tabular datasets. A 20,000 compute hours hyperparameter search for each learner. Every point of that search released. Leo Grinsztajn and his co-authors funded the comparison that way in 2022 so that no model could later be said to have lost on a tuning budget. Under that budget, tree-based models such as XGBoost and Random Forests remained state-of-the-art on medium-sized data of about 10K samples.

The abstract opens with the sentence the rest of the paper spends 45 datasets defending: “While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear.” The finding is stated just as plainly — “tree-based models remain state-of-the-art on medium-sized data” — and the qualifier of about 10K samples is part of the finding, not a hedge attached to it.

Read the result at its actual size. It is not that deep networks fail on tables. It is that at roughly that data size, once both sides were given 20,000 compute hours, the case for the more complicated family was not made. That is the shape of every honest escalation decision: a named data regime, a funded comparison, and a result that holds only inside the regime it was measured in.

Visual

Four families of intervention

The same project can contain several bottlenecks, but each technique should have one primary job.

Two of these four families already have measured answers attached to them, and the answers point in opposite directions. Take reuse first. ImageNet transfer was evaluated on two large-scale medical imaging tasks in 2019 by Maithra Raghu and three co-authors, in the study they called Transfusion. What they reported: “A performance evaluation on two large scale medical imaging tasks shows that surprisingly, transfer offers little benefit to performance, and simple, lightweight models can perform comparably to ImageNet architectures.” Reuse is a family of mechanisms, not a guarantee of gain.

For created supervision the number runs the other way. SimCLR, reported in 2020 by Ting Chen and three co-authors, learned representations with no labels at all; a linear classifier on top reached 76.5% top-1 ImageNet accuracy. That was a 7% relative improvement on the previous state of the art, and a match for a supervised ResNet-50. And, in the authors' words, “When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.” The family did not decide which of those two outcomes you get. The bottleneck did — labels were scarce in one case and not the binding constraint in the other.

FigureHierarchy · 4 levels

  • Combine predictions

    Use diversity across models to reduce variance, correct complementary errors, or stabilize a decision.

    • Reuse knowledge

      Transfer representations, parameters, examples, or adaptation strategies from previous learning.

      • Create supervision

        Derive useful targets from unlabeled data, weak signals, demonstrations, or logged behavior.

        • Change the learning regime

          Handle sequential tasks, graphs, sparse routing, offline interaction, or multi-task objectives.

The family tells you what kind of signal or structure the method adds.

Comparison

Baseline, advanced method, or system redesign?

Not every hard problem needs a more sophisticated learner. Sometimes the correct intervention is upstream or operational.

The first of the three columns has the largest evidence base behind it. That is the 19-algorithm, 176-dataset comparison again, and this is what it concluded: “we conduct the largest tabular data analysis to date, comparing 19 algorithms across 176 datasets, and we find that the 'NN vs. GBDT' debate is overemphasized: for a surprisingly high number of datasets, either the performance difference between GBDTs and NNs is negligible, or light hyperparameter tuning on a GBDT is more important than choosing between NNs and GBDTs.”

Take that as an instruction about sequencing rather than about tabular data. Light tuning on the existing learner is not the chore a team does while waiting for the interesting work to start. Across 176 datasets it frequently was the work. And the escalation it made unnecessary would have arrived with its own monitoring, its own retraining schedule and its own owner.

FigureComparison · 3 columns

Strengthen the baseline

Improve labels, splits, features, thresholds, or error analysis before changing the method.

  • Use when the failure is poorly understood
  • Lowest implementation and monitoring burden
  • Often exposes leakage or target problems
  • Example: fixing a time split beats adding a larger model

Add one advanced technique

Introduce a method with a mechanism tied to a specific limitation.

  • Use when the bottleneck is stable and measurable
  • Requires an ablation against the baseline
  • Adds method-specific failure modes
  • Example: bagging an unstable tree learner

Redesign the system

Change data collection, feedback, human review, or product constraints.

  • Use when the model is not the main bottleneck
  • May produce the largest real-world gain
  • Requires cross-functional ownership
  • Example: collect delayed labels instead of guessing proxies

Key idea

Complexity has to be budgeted

An advanced technique consumes more than compute. It also eats into the attention a team has for evaluating anything else. Stacking needs leak-free out-of-fold predictions. Domain adaptation needs target-domain validation. Continual learning requires tests for retained skills, not only the newest task. Mixture-of-experts introduces routing and load-balancing failures that a dense model does not have.

That last item is not an outside critic's complaint. It is in the abstract of the paper that brought sparse experts into the mainstream. Switch Transformers, published in 2022, describes the state of its own field this way: “However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability.” The same abstract reports up to 7x pre-training speedups over T5-Base and T5-Large. Both halves are true at once, and the second half is the one a team signs up to maintain. Later, DeepSeek described a mixture with 671B total parameters and 37B activated for each token that needed a purpose-built auxiliary-loss-free strategy for load balancing — an entire mechanism that exists only because the architecture does.

Before approving the method, list the new things that can go wrong and who will detect them. If the team cannot name the monitoring plan, the rollback path and the baseline comparison, it is not ready to own the technique.

The true price of complexity is the number of new assumptions you must test and maintain.

Case

The prize-winning ensemble Netflix declined to deploy

Netflix priced the engineering cost of an ensemble in public, and the bill was published twice: once by the engineering team in April 2012, and once by the engineer who inherited the system.

The prize offered $1 million for a 10% improvement on the Cinematch recommender. The first Progress Prize went to the Korbell team for an 8.43% improvement. Getting there took more than 2,000 hours of work and produced a final combination of 107 algorithms. That solution did ship, in part. Netflix took two of those algorithms into production and stopped there — the SVD variation with an RMSE of 0.8914 and the RBM at 0.8990, whose linear blend reached 0.88.

The $1 million Grand Prize went to something else: an ensemble of 104 individual predictors combined by a single-layer neural network. That one was never deployed at all. The engineering blog gave the reason: “the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment”. Xavier Amatriain, who ran the Netflix team that owned the rating-prediction algorithm, recorded the same decision in the first person in 2021: “But, what about the grand Prize solution with the 104 algorithms? As I mentioned in a blog post back then, we decided it was not even worth to productionize. It would have taken a large engineering effort for a small gain in accuracy that was most likely not worth it for several reasons.”

Two numbers, two fates. 107 algorithms in the Progress Prize combination, two of which reached production; 104 predictors in the ensemble that won the money and reached nothing. The accuracy was real in both cases. Only one of them was ever owned.

Figure

What an ensemble cost and how much of it shipped, beside what one tabular benchmark cost to run — the hours an advanced method spends that never appear in its score.

Example

Six bottlenecks and the techniques they suggest

These pairings are starting hypotheses, not automatic prescriptions.

  • Unstable tabular model: bagging or randomization may help when small sample changes produce large prediction swings — though on the 45-dataset benchmark, after 20,000 compute hours of search per learner, it was tree-based models such as XGBoost and Random Forests that stayed state-of-the-art at about 10K samples.
  • Scarce labels but abundant raw data: self-supervised pretraining can learn reusable representations before supervised fine-tuning. SimCLR reached 76.5% top-1 ImageNet accuracy with a linear classifier on its representations, matching a supervised ResNet-50, and 85.8% top-5 after fine-tuning on 1% of the labels.
  • New hospital or camera domain: domain adaptation may help if the target distribution is observable and sufficiently related to the source. The negative result is documented too — the 2019 Transfusion study found that “A performance evaluation on two large scale medical imaging tasks shows that surprisingly, transfer offers little benefit to performance, and simple, lightweight models can perform comparably to ImageNet architectures.”
  • Policy learning from historical logs: offline RL or imitation learning may be safer than online exploration, but support mismatch becomes central.
  • Fraud network with shared devices and accounts: a graph model can represent dependencies that row-wise features only approximate.
  • Many related tasks with uneven traffic: multi-task learning or sparse experts may share statistical strength while preserving specialization — Switch Transformers reported up to 7x pre-training speedups over T5-Base and T5-Large, and DeepSeek's 671B-total, 37B-activated-per-token mixture needed an auxiliary-loss-free strategy for load balancing to hold together.

Steps

The six-question escalation review

Use this review before the first advanced experiment, and again before production approval.

Step three is where these reviews usually fail. A counterfactual chosen after the fact is a formality; a counterfactual chosen in advance is a risk. Google Research supplied the demonstration in 2020. It revisited neural collaborative filtering and showed that with proper hyperparameter selection a simple dot product substantially outperforms the learned MLP similarities. Nothing about the advanced method changed between the original result and that one. The comparison did. That is why step three has to be answered while the method is still a proposal.

FigureProcess · 6 steps

  1. 1. State the failure

    Describe the slice, behavior, or constraint that the baseline cannot handle.

  2. 2. Identify the mechanism

    Explain why the candidate method should change that failure rather than merely add capacity.

  3. 3. Set the counterfactual

    Choose the strongest simpler alternative the method must beat.

  4. 4. Define success and harm

    Specify primary metrics, slices, compute cost, latency, and regression limits.

  5. 5. Run an ablation

    Change one advanced ingredient at a time whenever feasible.

  6. 6. Decide ownership

    Assign monitoring, retraining, rollback, and documentation responsibilities.

Escalation is a governance decision

Choosing an advanced method changes who must understand the system and what evidence must be retained. A routing layer, a replay buffer, a teacher model, a graph pipeline or a policy learner creates dependencies that survive the original experiment.

In at least one industry this has stopped being a matter of team preference. On 17 April 2026 the Federal Reserve Board, the FDIC and the OCC jointly issued revised model risk management guidance, replacing the guidance that had governed the question since 2011. The agencies say it is most relevant to banking organizations with over $30 billion in total assets, and that it may still apply to smaller ones with significant model risk.

Its governing principle is set out in the guidance itself: “Sound model risk management also involves “effective challenge,” which refers to the critical analysis conducted by objective experts who evaluate model risk and effect appropriate changes throughout the model lifecycle, from model development to ongoing monitoring.” The scrutiny is scaled to what the model can cost, with models of higher materiality that “warrant more comprehensive and rigorous oversight”. Objective experts, the whole lifecycle from development to monitoring, and a level of oversight set by materiality. That is the escalation review described in this lesson, written down by three regulators for the institutions they supervise.

Record the owner of each dependency, the monitoring signal that can reveal failure, and the fallback available when the component is removed. If those questions have no answer, the project has not budgeted its complexity.

Technical sophistication without an ownership model is deferred operational debt.

Steps

Write the escalation memo before choosing a method

A short decision memo forces the team to connect complexity to evidence.

Each of its five steps has a number behind it somewhere in this lesson. The counterfactual at step three is the finding, across 176 datasets, that light hyperparameter tuning on a gradient-boosted tree was often more important than the choice between model families. The price at step four is the auxiliary-loss-free load-balancing strategy that a 671B-total, 37B-activated-per-token mixture had to build for itself. The stopping rule at step five is the ensemble of 104 individual predictors that won $1 million and was judged not worth productionizing. Write the memo while those are still hypothetical costs. After the complicated thing exists, they are owned ones.

FigureProcess · 5 steps

  1. 1. Name the bottleneck

    Describe the observed failure using slices, examples, and operational consequences.

  2. 2. State the mechanism

    Explain why the proposed technique could change that failure rather than merely add capacity.

  3. 3. Define the counterfactual

    Choose the strongest simpler baseline that would falsify the need for the advanced method.

  4. 4. Price the complexity

    List new data, compute, monitoring, and ownership obligations.

  5. 5. Set a stopping rule

    Specify the evidence that would end the experiment or trigger a rollback.

Position

Half of every reported gain belongs to the baseline

Read any reported gain and you are reading two models, not one: the method, and whatever it was measured against. Nobody's reputation rests on how well that second model was built. It is also the part of the experiment standing between a team and its result, so it tends to receive whatever tuning budget is left at the end. In 2019 Maurizio Ferrari Dacrema and two co-authors counted what that costs a literature: “Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques.” Eighteen published advances. Seven that could be reproduced with reasonable effort. Six of those seven beaten by nearest-neighbour or graph-based heuristics. A year later Google Research showed the same thing on a single method: with proper hyperparameter selection, a simple dot product substantially outperforms the learned MLP similarities of neural collaborative filtering.

One benchmark removed that asymmetry by paying for it. Forty-five datasets, and every learner — trees and deep networks alike — given a 20,000 compute hours hyperparameter search, every point of which was released. Under that budget the tree-based models such as XGBoost and Random Forests remained state-of-the-art on medium-sized data of about 10K samples. Its abstract opens with the reason the study was worth running at all: “While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear.” Read the finding at its actual size. It is not that deep networks fail on tables. It is that at roughly that data size, once the comparison was funded properly, the case for them was not made.

The second number a headline omits is the one Netflix published. A 10% improvement on Cinematch was worth $1 million. The first Progress Prize went to a combination of 107 algorithms and more than 2,000 hours of work, of which two algorithms reached production. The Grand Prize ensemble of 104 individual predictors reached none. Xavier Amatriain, who led the team that owned the rating-prediction algorithm, wrote in 2021: “But, what about the grand Prize solution with the 104 algorithms? As I mentioned in a blog post back then, we decided it was not even worth to productionize. It would have taken a large engineering effort for a small gain in accuracy that was most likely not worth it for several reasons.” A measured improvement and a deployed improvement are separate claims. Only the second arrives with an owner, a monitoring signal and a rollback path. That is why the escalation review here sets the counterfactual at step three, while the method is still a proposal: the strongest simple alternative is easy to name in advance and nearly impossible to name honestly once somebody has built the complicated thing.

Half of a reported gain is a statement about the baseline, and the baseline is somebody’s choice.

Key takeaways