Weak Supervision: Combining Noisy Sources Instead of Waiting for Perfect Labels

A team needs training labels for millions of items and has only a small expert-reviewed set. Google shipped exactly that system and published what it cost, so there is no need to imagine one.

Snorkel DryBell produced training labels for three production classifiers. Google's own blog post about the deployment, in March 2019, gives the result: “we achieved the same predictive accuracy training a model with Snorkel DryBell's labels as we did when training that same model with 12,000 and 80,000 hand-labeled training data points.” The case study behind that post went to SIGMOD in 2019, written by a team from Google, Brown and Stanford.

On the first classifier one developer wrote ten labeling functions over 684,000 unlabelled data points. None of the sources was an oracle. They were heuristics over the linked URL, an internal named-entity tagger, and an internal topic model far too coarse for the task but usable as a negative signal. The model trained on their combined output “matched the performance of 80K hand-labeled training labels”, and landed within 4.6 F1 points of a model trained on 175,000. On the second task eight labeling functions matched 12,000 hand labels. Across the three classifiers, converting non-servable organisational resources into servable models gave “an average 52% performance improvement”. Ten rules, one developer, no new annotation budget.

Instead of labeling every example manually, experts write or select sources that label many examples imperfectly, and those sources may be heuristics, patterns, knowledge bases, existing models, partial annotations, or distant matches.

The resulting labels are noisy. They are often correlated too. The project therefore models or manages source quality. It does not pretend each source is an independent oracle.

The oldest industrial-scale version of this states its own error rate in the abstract. In 2009 Mintz and colleagues filtered Freebase down to “1.8 million instances of 102 relations connecting 940,000 entities”. They then labelled training data by a rule that is frequently wrong. Any sentence in a large Wikipedia corpus containing both members of a known pair was treated as expressing that relation. Two entities can share a sentence for any reason at all. With no hand-labelled corpus, their classifier was “able to extract 10,000 instances of 102 relations at a precision of 67.6%”. About two-thirds right, at a scale no annotation budget of the period could have reached.

Weak supervision separates source creation from final model training. The DryBell deployment is one instance of this pipeline: sources first, then combination, then a single end model trained on the combined output and judged against reviewed examples.

The weakness can come from noise, incompleteness, or indirect connection to the target, and the three-way split is not this lesson's invention. Zhi-Hua Zhou fixed it in a 2018 review: incomplete supervision, where only a subset of the training data carries labels; inexact supervision, where only coarse-grained labels are given; and inaccurate supervision, where the given labels are not always ground truth. Other fields adopted the classification and cited it. A 2020 paper on weakly supervised segmentation of satellite images opens its definition section with “In his excellent review, (Zhou, 2018) defines weakly supervised learning as an umbrella term addressing the attempt to construct predictive models from three types of weak supervision:” Then it narrows the term, keeping the inexact and inaccurate cases for itself and treating incomplete supervision as semi-supervised learning. A taxonomy that a second field can argue with is a taxonomy, not a slogan.

Distant supervision earns its column by writing the error into its own premise. The 2009 rule is stated without ceremony: “If two entities participate in a relation, any sentence that contains those two entities might express that relation.” A survey by Smirnova and Cudré-Mauroux numbers it Assumption 3.1 and shows what it costs. Two sentences contain Elon Musk and Tesla. Only one of them expresses the relation the knowledge base records. The second sentence becomes a labelled training example anyway. Riedel and colleagues proposed a weaker replacement in 2010, quoted in the same survey as Assumption 4.1: at least one sentence in the bag expresses the relation, rather than any of them.

Partial supervision is a production reality, and its owners document the gap. Open Images V4 reports 30.1M image-level labels for 19.8k concepts, and 15.4M bounding boxes on 1.9M images. Google's own V7 documentation records 61.4M human-verified image-level labels across 20,638 classes, alongside 164.8M machine-generated training labels. It also says what verification does and does not do: it removes false positives, not false negatives. The consequence is stated in one sentence. “All other classes which are not explicitly marked as positive or negative for an image are not annotated.” Absence in that file is not evidence of absence in the image.

The sources are part of the model and deserve versioning, tests, and retirement rules. Step 2 is where the honesty of the whole system is decided, because the small trusted set is not neutral scaffolding: it is labelled data you already own, and it can be spent on the end model directly.

How to spend it is the subject of a 2023 paper that took a Theme Paper Award at ACL. Its abstract states the finding: “Specifically, we find that the success of existing weakly supervised learning approaches heavily relies on the availability of clean validation samples which, as we show, can be leveraged much more efficiently by simply training on them.” The reported benefits of weak supervision were significantly overestimated, its authors argue, because the clean validation set used to select the model could have been training data. This held even at five clean samples per class.

BOXWRENCH answered in January 2025. On the existing WRENCH datasets it replicated the result, confirming crossover points below 200 clean labels for four of six tasks. On its own harder, higher-cardinality tasks the crossover moved beyond 1,000 clean labels. Step 5 therefore has a real number attached, and it is a range rather than a rule. Somewhere between fewer than 200 and more than 1,000 hand labels, direct annotation overtakes the weak-label pipeline. Which end of that range you are on depends on how hard your task is.

Expert effort moves from row-by-row annotation to source design, gold-set creation, conflict analysis, and maintenance, and the shift can be highly efficient when domain rules cover many cases.

It is less attractive when the target is poorly defined, sources have low coverage, or systematic bias is difficult to observe. Sometimes direct labeling or active learning is simpler.

The residual cost is measurable, and the standard benchmark measures it. WRENCH, published at NeurIPS in 2021, collects 22 real-world datasets for classification and sequence tagging and compares more than 120 method variants on them. On its eight sequence-tagging datasets with a BERT end model, training on gold labels averaged 84.36 F1. The best weak-supervision configuration, the CHMM label model, averaged 66.50. The gap is roughly 18 points. The paper does not soften it. “As a result, there is still a significant performance gap between the results trained by gold labels and weak labels, which motivates the future research on designing methods robust against the induced noise.”

The benchmark's other recommendation is deflating for the machinery around the labels: plain majority vote is among the best label models for classification, a takeaway the BOXWRENCH group adopted as its own default in 2025. So the sophistication has to earn its place against the simplest baseline, and the expert work has to be spent where it counts. In the DryBell deployment it did not vanish. It became eighteen labeling functions across the two tasks and a set of judgements about which internal resources deserved trust.