Semantics: Meaning, Ambiguity, and Lexical Relations

Failures can arise at any layer even when the words are familiar, and each layer in this lesson has a published failure attached to it. Lexical sense: the Most Frequent Sense heuristic reads no context at all and still reaches 64.8 F1 on the ALL concatenation of the unified word-sense evaluation. Compositional structure: the 100% to 0.0% flip Ettinger measured when a single negation was inserted. Reference and entities: the 44,000 pronoun problems of WinoGrande, where systems sit 15 to 35 points under the 94.0% human score. Discourse and context: the entailment pairs of SNLI, two-thirds of which can be labelled without reading the premise sentence at all. World and domain knowledge: a missing comma in a Maine statute, which a federal appeals court had to interpret because the text alone did not settle it.

The layers are not a hierarchy of difficulty. A system can be strong at one and indifferent to another. An aggregate accuracy number mixes them together.

A score or embedding should not be described generically as “meaning.” Similarity is graded and often symmetric. Entailment is directional and sensitive to quantifiers and negation. Contradiction asks whether two statements can hold in the same interpretation. Relatedness is an associative signal, useful for discovery and weak for substitution.

The distinctions are not academic, because a benchmark built for one of them can be passed by a model that computed another. The two sections that follow are both instances of that: an entailment corpus whose hypotheses leaked their labels, and a sense task whose commonest answer is available without reading the sentence. Naming the relation your product needs is what makes those failures visible.

A thesaurus-like expansion needs finer controls than “nearby word.”

These relations have been a queryable database for three decades. WordNet is “an online lexical database designed for use under program control”, as George A. Miller put it in 1995. He stated the design plainly: “English nouns, verbs, adjectives, and adverbs are organized into sets of synonyms, each representing a lexicalized concept”, and “semantic relations link the synonym sets”. His motivation was that “dictionary entries evolved for the convenience of human readers, not for machines”. The unit is the synonym set, not the wordform. That is what makes hypernymy expressible at all — and what makes nearby word the wrong query to send to a lexical resource.

Having the inventory is not the same as choosing correctly within it. In the unified word-sense disambiguation framework published in 2017, the Most Frequent Sense heuristic — which reads no context whatsoever — scores 64.8 F1 on the ALL concatenation, 67.6 on nouns and 49.6 on verbs. The best supervised system trained on SemCor scores 69.6. The framework's own authors diagnose why: “all supervised systems suffer a strong bias towards the MFS, with all IMS-based systems disambiguating over 75% of instances with their MFS”. Blevins and Zettlemoyer reproduced the baselines in 2020 — 65.5 F1 for MFS, 65.2 for the WordNet first sense on ALL — and reported the split that matters. The WordNet-first-sense baseline scores 100.0 on most-frequent-sense instances and 0.0 on less-frequent-sense instances. A product's domain senses live in that 0.0.

An event description can distinguish who performed an action, what changed, what instrument was used, where it occurred, and when. Active and passive sentences may express similar roles through different syntax. The canonical demonstration is a pair of sentences: “John broke the window” and “The window broke”. The window occupies two different syntactic positions and fills the same underlying role in both.

The answer to that was PropBank, a predicate–argument layer added over the Penn Treebank Wall Street Journal text. Palmer and colleagues built it and described it in 2005. The scale is on record: the release Proposition Bank I, dated 1 September 2004, contains approximately 113,000 annotated verb tokens and annotations for over 3,200 unique verbs. So is the boundary of what a role layer buys you. The authors state it themselves in their abstract: “The resulting resource can be thought of as shallow, in that it does not represent coreference, quantification, and many other higher-order phenomena, but also broad, in that it covers every instance of every verb in the corpus and allows representative statistics to be calculated.”

Read that sentence as a specification rather than a caveat. Broad coverage over one text type, with coreference and quantification deliberately out of scope, is a design choice made for a purpose. It is the same choice a product team makes when it decides which participants its workflow actually needs. Define those, then test omitted, implicit, and coordinated arguments against them.

Use controlled contrasts so that each example tests one meaning operation. There is a worked precedent with published numbers. HANS is a 30,000-example controlled evaluation set: 1,000 examples per template, 10,000 per heuristic, for three heuristics a model might apply instead of computing entailment. McCoy and colleagues published it in 2019.

The result splits cleanly by answer. Models trained on MNLI, BERT included, were near-perfect when the answer was entailment and near-zero when it was non-entailment. The paper states it: “All models performed substantially below chance on this dataset, barely exceeding 0% accuracy in most cases.” Clark and colleagues independently measured BERT at 62.40 on HANS against 84.24 on the MNLI matched dev set.

Notice what that gap is made of. The in-distribution test set reported 84.24 and never showed the problem, because the shortcut and the relation agree on most naturally occurring pairs. The templated contrast set forced them apart. The five steps below are the same construction discipline: vary one thing, keep the rest fixed, and make sure both answers are represented.

Take ten query–document pairs labeled “relevant.” For each pair, state whether relevance comes from synonymy, hierarchy, event match, question answer, shared topic, or an external fact.

Then add one hard negative that is topically similar but contradicts or fails to support the query, and measure whether the ranking system preserves the distinction. This is the non-entailment half of HANS in miniature, and it is the half where the reported aggregate number is silent.

Run the Gururangan check as well: score your labels with half the input removed. If a classifier that never sees the document can reproduce your relevance labels at well above the majority-class rate, the labels encode how they were written rather than the relation you meant to test.