Advanced Techniques Capstone

A global marketplace wants one platform to support fraud detection, seller-quality prediction, recommendation ranking, and deciding which cases investigators open first. Labels are delayed and uneven across regions. New sellers have little history. Accounts, devices, payments, and merchants form a graph. A large pretrained encoder exists, and the platform also holds a much larger pool of unlabeled activity. Production latency is strict, policies change over time, and investigators can review only a limited queue.

Every advanced method in this path is tempting here. That would be a design failure. The task is to choose a minimal sequence of interventions, each tied to evidence and a matched counterfactual.

The brief itself is an exercise; nothing in it is a finding. Every gate imposed on it below comes from a published measurement, and each one is named where it applies: a 47,000-run graph re-evaluation, a reproducibility audit of 18 neural recommenders, a benchmark of ten real distribution shifts, two distillation results, one company's decision to shut an algorithmic business down, and one federal framework's numbered requirement that somebody be able to switch a model off.

The project contains several bottlenecks, but they should be addressed in an explicit order.

Each branch below is a hypothesis, and each already has a published measurement attached to it. The relational branch answers to more than 47,000 controlled runs that compared five graph neural networks against baselines ignoring the edges. The changing-behavior and regional branches answer to WILDS, where standard training loses 93.2% to 70.3% average accuracy across hospitals on Camelyon17. The serving branch answers to two distillation results, one from 2006 and one from 2015. The sections below set all of them out. The map's only job is to say which one is being tested.

Each strategy solves a different subset of the problem and creates different ownership costs.

Read the three rows against the re-evaluation literature, not against their own promises. The conservative modular system is the option that beat most of the neural recommenders the 2019 reproducibility audit could reproduce at all. It is also the option that beat learned similarities in Google's own follow-up, once hyperparameters were properly selected. The graph and routed architecture is the option that failed to beat structure-agnostic baselines on several of nine benchmarks in the 47,000-run comparison. Neither result says the ambitious row is wrong here. Both say it has to be measured against the modest row before it is built.

The review should force claims to become measurable commitments.

The reason the first bullet is first: a benchmark called WILDS collects ten datasets whose splits follow real boundaries — hospital, camera, country, demographic group, scaffold, time and region, repository, user, location, batch. Standard models were trained on all ten. Camelyon17 falls from 93.2% to 70.3% average accuracy across hospitals. CivilComments falls from 92.2% to 56.0% worst-group accuracy. FMoW falls from 48.6% to 32.3% worst-region accuracy. The 2021 summary is flat: “On each dataset, we show that standard training yields substantially lower out-of-distribution than in-distribution performance. This gap remains even with models trained by existing methods for tackling distribution shifts, underscoring the need for new methods for training models that are more robust to the types of distribution shifts that arise in practice.” CORAL, IRM and Group DRO are the methods a design review would reach for on being shown those gaps. They generally failed to improve on standard empirical risk minimisation. A marketplace whose labels are uneven across regions is asking for exactly this measurement, and no aggregate score will supply it.

Later rows should not begin until earlier evidence passes the agreed gate. The order is not fastidiousness. Each step is here because a published re-evaluation found that step deciding the result the step below it was given credit for.

Step 1 is the split, and the split is often the whole finding. The WILDS collapses above were measured across hospitals, regions and demographic groups, never across randomly shuffled rows.

Step 2 is the one most often skipped, and the one with the largest recorded body count. Ferrari Dacrema and two colleagues took 18 neural top-n recommendation methods published at top conferences and tried to reproduce them at RecSys 2019: “Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques.” The next year Google's own paper reached the same verdict from the other direction, setting neural collaborative filtering against plain matrix factorization: “With a proper hyperparameter selection, a simple dot product substantially outperforms the proposed learned similarities”. An untuned baseline is not a baseline. It is a foil, and a design review that accepts one has already decided the outcome.

Step 4 asks whether the edges carry signal at all. That is a separate question from whether a graph network is good at using them. Errica and three colleagues answered it at scale in 2020: “To counter this troubling trend, we ran more than 47000 experiments in a controlled and uniform framework to re-evaluate five popular models across nine common benchmarks. Moreover, by comparing GNNs with structure-agnostic baselines we provide convincing evidence that, on some datasets, structural information has not been exploited yet.” A model that ignores the edges entirely was competitive on several of those nine datasets. Run that model first here, on the marketplace's own temporal and new-node splits. The graph platform then either earns its ownership cost or is dropped in a week.

Step 6 prices the serving constraint, and here the evidence is unusually kind. In 2006 three researchers at Cornell compressed large ensembles into single networks and concluded: “Results on eight test problems show that, on average, the loss in performance due to compression is usually negligible, yet the mimic neural nets are 1000 times smaller and 1000 times faster.” Geoffrey Hinton and two colleagues reported the same trade with production numbers in 2015. On about 2,000 hours of spoken English, roughly 700M training examples, a 10-model acoustic-model ensemble reached 61.1% test frame accuracy and 10.7% WER. A single distilled model reached 60.8% and 10.7%. The single-model baseline reached 58.9% and 10.9%. On MNIST the same paper counts 67 errors for the large regularised net, 146 for the unregularised small net, and 74 for that same small net distilled at temperature 20. Distillation buys the latency budget cheaply — but only after steps 2 to 5 have established that the teacher is worth copying.

Each advanced component enters only after a simpler experiment establishes the bottleneck it is meant to remove. Transfer follows a target-only baseline. Graph learning follows relational features. Sharing follows stable single-task models.

This sequence makes negative results useful. When an intervention fails its gate, the team can stop without invalidating the rest of the system or hiding the result inside an end-to-end score.

The failure it is defending against has a name: underspecification. A pipeline is underspecified when it can return “many predictors with equivalently strong held-out performance in the training domain” — and “such predictors can behave very differently in deployment domains”. D’Amour and his co-authors set that out in 2020. The held-out score cannot choose between such predictors, because by construction they tie on it. The stress test has to.

How much room a tie leaves is measurable. In 2019 Recht and three colleagues built new test sets that replicated the original CIFAR-10 and ImageNet collection procedures, then re-scored a wide range of published classifiers. Accuracy dropped by 3–15% on CIFAR-10 and by 11–14% on ImageNet. The new sets were built to the same recipe, with no shift anyone intended. The ordering largely held, since “accuracy gains on the original test sets translate to larger gains on the new test sets”, but the level did not. A gate that reads a single held-out number to two decimal places is reading past the noise floor of its own benchmark.

A production-ready proposal should contain more than architecture diagrams.

The ownership sections are not boilerplate, and they are not left to taste. The U.S. National Institute of Standards and Technology released its AI Risk Management Framework on 26 January 2023. The framework is voluntary. It still states the requirement as an outcome to be achieved rather than an aspiration. Its MANAGE function includes: “MANAGE 2.4: Mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use.” MANAGE 4.1 adds that post-deployment monitoring plans are implemented, including mechanisms for appeal and override, decommissioning, incident response, recovery and change management. A package that cannot say which named team executes each of those verbs has not been finished, whatever its offline scores are.

The advanced practitioner is not the person who can name the most methods. It is the person who can identify the narrow mechanism a project needs, build a fair comparison, and stop when the evidence is weak.

The cost of the alternative is on the record, and it is specific. More than 47,000 controlled runs were needed before anyone could say whether five popular graph networks beat models that ignore the edges. Of 18 published neural recommenders, 7 could be reproduced with reasonable effort, and 6 of those 7 were often beaten by nearest-neighbour or graph heuristics. Camelyon17 scores 93.2% in distribution and 70.3% across hospitals. And on 2 November 2021 a company took an approximately $304 million write-down and cut approximately 25% of its workforce to stop a model-driven business it had already scaled.

Ensembles, transfer, self-supervision, meta-learning, continual learning, reinforcement learning, graphs, and sparse experts are powerful because they change where learning signal or capacity comes from. That same change creates new failure modes. The course ends where production judgment begins: every technique is a hypothesis, every hypothesis needs a counterfactual, and every deployed component needs an owner who, in the words of MANAGE 2.4, can supersede, disengage, or deactivate it.