Pragmatics, Discourse, and Coreference

The literal sentence form does not always identify the communicative action. Each line below is grammatical, and in each the thing being done sits somewhere other than in the surface syntax — in what the utterance assumes, in what it leaves out, in who is authorized to say it.

A conversation system needs more than the last user message.

One corpus tried to hold several of these layers at once, and it shows how unevenly they fill. OntoNotes Release 5.0 layers “structural information (syntax and predicate argument structure) and shallow semantics (word sense linked to an ontology and coreference)” over three languages: English, Chinese and Arabic. It spans several genres too — newswire, broadcast news, broadcast conversation, web data and telephone speech. The release description gives the size. “This cumulative publication consists of 2.9 million words”.

The Arabic portion is the exception that prices the rest. It is 300,000 words, and the per-genre table carries it under News only: no broadcast news, no broadcast conversation, no web, no telephone. Discourse state is expensive to annotate. The budget shows up as coverage rather than as a line item. What goes missing is the languages and the genres outside the newswire.

Reference can require syntax, discourse, perception, and world knowledge. Each of those has been measured, and the measurements are not close together.

The distinction was built into a shared task rather than left to intuition. Pradhan and colleagues describe it: “The CoNLL-2012 shared task involved predicting coreference in English, Chinese, and Arabic, using the final version, v5.0, of the OntoNotes corpus”. That corpus, they write, “provides a large-scale corpus of general anaphoric coreference not restricted to noun phrases or to a specified set of entity types” — unlike the earlier MUC and ACE corpora. Widening the definition is what made the harder cases countable.

Start with the easiest column below: plain identity, “Dr. Lee … she”. Google AI Language built a corpus for exactly that case. GAP is gender-balanced, holds 8,908 ambiguous pronoun-name pairs sampled from Wikipedia, and appeared in Transactions of the ACL in 2018. Its abstract does not describe a solved problem: “We explore a range of baselines that demonstrate the complexity of the challenge, the best achieving just 66.9% F1.” A shared task on that corpus the following year drew 263 competing teams, and the winning system reached 0.13667 logloss with near gender parity. Two thirds on the simplest case, after a field-wide effort. That is the baseline against which everything else here should be read.

Then ask what the “world knowledge” in the fourth column consists of. WinoBias put three coreference systems of three different designs through the same test. Zhao and colleagues report what came back: “We demonstrate that a rule-based, a feature-rich, and a neural coreference system all link gendered pronouns to pro-stereotypical entities with higher accuracy than anti-stereotypical entities, by an average difference of 21.1 in F1 score.” Rudinger and colleagues at Johns Hopkins built the Winogender schemas the same year — minimal pairs differing only in the gender of the pronoun — and confirmed systematic gender bias in three publicly available coreference systems. Two groups, two test sets, one finding. A large part of the knowledge these systems bring to reference is occupational stereotype, and it is worth 21.1 F1.

The hard end of reference was designed as a commonsense test and then stopped being one. The Winograd Schema Challenge was proposed by Hector Levesque as an alternative to the Turing test. It ran as a competition exactly once, at IJCAI-16 on 11 July 2016, over a test collection of 60 pronoun disambiguation problems. Kocijan and colleagues record the outcome: “The most successful of these was that of Liu et al. (2017b) discussed in Appendix B.3, which achieved a score of 58% on the test collection.” No prize was awarded, and the contest never reached its second round.

Then the ground moved. “By 2019, a number of AI systems, based on large pre-trained transformer-based language models and fine-tuned on these kinds of problems, achieved better than 90% accuracy.” The WinoGrande paper records 90.1% on “a set of 273 expert-crafted pronoun resolution problems”, against human performance of 94.0%. Whether what carried a score from 58% to 90.1% is commonsense or dataset regularity is precisely the question a lesson on reference should leave open.

Discourse relations can include cause, contrast, elaboration, evidence, condition, and sequence; headings, connectives, tense, entity continuity, and genre conventions help readers build a coherent interpretation.

That list is not a taxonomy someone sketched. It has been annotated and counted. The Penn Discourse Treebank marks discourse relations over “the Wall Street Journal (WSJ) section of Treebank-2 (LDC95T7)”. Version 2.0, from 2008, carried over 40,000 annotated relation tokens; the distributor's own record puts the figure carried forward at 40,600. Prasad and colleagues then announced the next round: “PDTB-3, which we plan to release to the community in Fall 2018, will contain over 53K tokens of discourse relations”. Their abstract says where the roughly 13,000 additional relations were found: “We have increased its coverage of discourse relations (from ~40K to ~53k), the majority in intra-sentential contexts”. PDTB-3 was released on 15 March 2019.

Read that increase carefully, because it is the practical lesson. Thirteen thousand relations were sitting in a corpus that had already been annotated once, inside sentences the earlier pass had treated as indivisible. Coherence is not only the connective tissue between sentences. Most of the relational structure in that newspaper prose lay below the sentence boundary, where nobody had been looking.

A summary or answer can preserve true sentences while destroying these relations. Evaluate whether causal direction, contrast, chronology, attribution, and uncertainty survive compression or generation. And note that a system rewriting within a sentence is operating exactly where PDTB-3 found most of its new relations.

When uncertainty affects the next action, the system should narrow it rather than guess silently. That is not a design preference imported from etiquette. It is how human conversation already runs, at a rate that has been measured.

The measurement covers 48.5 exhaustively sampled hours of spontaneous conversation, in 12 languages from 8 language families. Dingemanse and eleven colleagues coded all of it and found 2,053 repair sequences. Their result holds right across the sample: “there is a repair initiation about once every 1.4 minutes across all languages” (PLOS ONE, 2015). A human-robot interaction workshop later restated the same rate in seconds, saying of the utterance “huh?” that it “occurs around once every 84 s in conversation and appears to be universal across human languages”.

Once every 84 seconds, in every language sampled, someone stops the exchange to check what was meant. A system that never asks is therefore not the efficient participant in the conversation. It is the outlier. The steps below describe what asking well looks like: detecting the alternatives, pricing the mistake, asking the smallest useful question, recording the correction, and leaving the person a way out.

Turn-level accuracy can miss whether the system remembered a constraint, repaired an error, or completed the user's goal; build scenarios with reference chains, corrections, topic changes, interruptions, and missing information.

There is a documented reason not to take a turn-level number on faith. MultiWOZ, the standard task-oriented dialogue benchmark of over 10,000 dialogues, carried enough annotation noise that two separate organisations re-annotated it. Amazon Alexa AI released MultiWOZ 2.1 in 2019 and reported the scale of the repair: “This correction process results in changes to over 32% of state annotations across 40% of the dialogue turns.” The following year Google audited the corrected corpus and found more still: “we identify and fix dialogue state annotation errors across 17.3% of the utterances on top of MultiWOZ 2.1”.

That is the field's main dialogue-state benchmark, and a third of its state labels were wrong. Every score published against it before those corrections was partly a score of the annotation rather than of the system. Then a second independent audit of the corrected version moved another 17.3% of utterances.

Measure state accuracy, successful clarification, unnecessary questioning, recovery, task completion, and final consequence; inspect whether errors persist across turns or are contained. And before trusting any of those numbers, ask who annotated the state and whether anyone has ever audited it.

First mark entities, mention clusters, dates, goals, open questions, and corrections; then annotate speech acts and discourse relations such as request, refusal, evidence, contrast, and repair.

Compare two annotators. Discuss which disagreements arise from ambiguous language, missing context, or an inadequate schema. Expect disagreement to be the normal outcome rather than the failure case, because it is the normal outcome for the professionals. The Penn Discourse Treebank found roughly 13,000 further relations on a second pass over text it had already annotated. Amazon Alexa AI changed over 32% of the state annotations in MultiWOZ, and Google then corrected another 17.3% of its utterances on top of that. Well-resourced projects disagree with their own earlier selves at those rates, over months of work, with published guidelines.

When your two annotators diverge, resist the reflex to name a winner. Write down which of the three causes produced the split. Then check whether the schema simply has no slot for what the speaker did — as the nine dimensions of ISO 24617-2:2012 had none for Contact Management or Task Management.