A Bounded Evaluation Protocol for Representational Systematicity

AI summary

Overview

Research area: Natural Language Processing / interpretability and compositional generalization (systematicity) in neural networks.

Technical level: Advanced. The paper is a methodology contribution: it defines a formal evaluation protocol with scoring functions, gates, and stop rules, and assumes familiarity with probing, activation patching, and interchange interventions.

Scope (one sentence): The paper proposes Bounded-RS, a protocol that adjudicates representation-level explanations of systematic behavior into four bounded evidential states, and demonstrates it on a 96-set semantic-parsing testbed with fine-tuned T5 models.

What This Paper Is About

Neural models can produce correct outputs for novel combinations of familiar parts, but that behavioral success does not show that the model actually represents the parts and their roles and uses them to generate its outputs. The paper argues that the usual kinds of evidence each mislead on their own—correct outputs can come from shortcuts, a decodable signal need not be used, an intervention can act through unintended pathways, and a failed search is not proof of absence—and that existing work leaves implicit which claim a given evidence pattern licenses. The goal is a fixed, falsifiable procedure that maps evidence about a declared explanation to one of four bounded verdicts, so that claims about different explanations can be compared.

Key Contributions

1. A four-state adjudication scheme. Bounded-RS assigns an explanation one of four states—Unsupported, Present-only, Causal-use, or Undetermined—each with a defined minimum evidence pattern and a stated licensed claim, so that a decodable-but-unused signal cannot be reported as used.

2. A layered protocol with frozen rules. A data layer fixes the behavioral contrast with matched examples; an evidence layer collects behavioral anchor, recoverability, interchange/necessity/repair interventions, and realism and item-linkage checks; a decision layer combines control-calibrated scores, gates, and stop rules whose parameters are fixed on calibration controls before target outcomes are interpreted.

3. A bounded-negative mechanism (closure). A negative about an entire explanation family is licensed only when every admissible object in a declared, frozen closure basis fails; extending the negative to an ontology-level claim additionally requires detectors validated as truthful monitors.

4. A released testbed and a two-family audit. A synthetic semantic-parsing testbed of 96 matched sets (672 items, 192 interchange pairs, 96 repair pairs) plus protocol code and audit reports, used to adjudicate a role–filler Binding explanation and a distributed Geometry explanation on T5-small and T5-base.

Main Findings

• Binding is Causal-use. The normally fine-tuned T5-small target, which reaches exact match and clause-level F1 of 1.0 on both held-out splits, receives Causal-use: its role signal is recoverable beyond surface controls, and its interchange, necessity, and repair effects exceed the strongest matched controls with realism and linkage gates passing.

• Geometry is Unsupported at the family level. After closure over a frozen basis of 36 objects in 3 classes, no admissible object passes the causal-use (necessity) gate, and no direct or slice-local Present-only reading holds. Only the cross-step bridge class (8 of 12 candidates surviving discovery) was accepted at discovery; interaction objects and query–source relations (4 of 12 survivors each) were not accepted, so no object of those classes could pass.

• The decision rule separates states in the target panel. The five-run Binding target audit returns 1 Causal-use, 1 Present-only, 1 Unsupported, and 2 Undetermined; the seven-run Geometry target audit returns 2 Causal-use, 1 Present-only, 3 Unsupported, and 1 Undetermined at the operation level, which the authors state cannot support a family-level claim on its own.

• Held-out control accuracy is imperfect, with Present-only the weak point. Binding held-out controls: 6 controls, accuracy 0.833, macro-F1 0.822, with one Present-only control escalated to Causal-use. Geometry held-out controls: 9 controls, accuracy 0.778, macro-F1 0.796, recovering only one of three Present-only controls.

• Verdicts are stable across configurations, seeds, and resampling. Both verdicts hold across 487 calibration-compatible configurations from a 729-configuration grid and across five training seeds; across 1,000 matched-set bootstrap replicates the Geometry family is Unsupported in 998 and Undetermined in 2, never Causal-use. For the first seed, the interchange advantage over the strongest control is 0.337 (95% matched-set bootstrap interval [0.301, 0.371]) against the 0.245 use boundary.

• The negative is not an artifact of the search. In engineered in-basis truth controls, the accepted cross-step branch returns Causal-use, Present-only, and Unsupported correctly in 3/3 seeds each, with a correct-object advantage of 0.335 above the 0.245 use boundary.

• The Geometry verdict stays family-final. None of the three detector classes is validated as a truthful monitor, each tested on 336 canonical examples, so the paper does not make an ontology-level negative.

• Full adjudication is more reliable than its component rules. On 41 predeclared cases with known causal status, taking recoverability as use licenses Causal-use falsely in 13 of 34 noncausal cases, ungated interventions in 7, and full Bounded-RS in 1, while retaining all 7 causal cases and abstaining on 6 of the 41.

• Verdicts transfer. On T5-base with the protocol and family interfaces unchanged, held-out controls are correct in 16 of 18 Binding and 22 of 27 Geometry cases, and all six seed-by-rule runs return Binding Causal-use and Geometry family-final Unsupported. On a prespecified 96-pair exact-contrast SLOG subdomain, the unchanged Binding chain returns Causal-use, Undetermined, and Causal-use across three T5-small seeds.

Methodology in Plain English

The authors start by building a controlled testbed rather than using natural text. Each of 96 matched sets is generated from one underlying meaning (a predicate with agent, theme, and modifiers) and rendered into seven sentences that each change exactly one thing: an easy active-voice source, a hard target in the set's construction, agent replacement, theme replacement, a role swap, a filler replacement, and a modifier swap. Because the items share gold meanings and differ in a single controlled way, the researchers can swap internal signals between matched items and insert a signal from an easy source into a hard target, then compare each effect against deliberately mismatched controls.

They fine-tune T5-small on 64 of the sets and hold out 32, including all six passive-with-relative-clause sets that never appear in training. They then treat two candidate explanations as separate families sharing the same testbed and decision rules but supplying their own detectors, interventions, and controls. The Binding family targets a role-indexed or aggregate role–filler signal; the Geometry family targets distributed objects such as predicate-conditioned relation subspaces, cross-step bridges, and query–source relations.

Before looking at any target result, they fix every scoring boundary, scale, weight, and gate using calibration controls built to instantiate each intended state. Evidence is rescaled onto a common sigmoid support scale, group scores for behavior, presence, and use are combined, and stop rules return Undetermined when coverage is at most 0.80, when the top state leads the next by no more than 0.10, or when required gates fail. Behavior is capped at a 4% share of decision weight, so accurate outputs alone cannot yield Causal-use. For the Geometry negative, they freeze a basis of object classes, seal observability with an aligner, run discovery, and then require every admissible object to fail the necessity gate before the family can be called Unsupported.

Why This Matters

Impact on research. The paper turns a long-running conceptual dispute—traceable to Fodor and Pylyshyn's argument that connectionist models lack structured representations—into comparable, falsifiable verdicts tied to declared explanations. It also changes reporting norms: a probing study would report Present-only until controlled interventions pass, and a negative study would state the search space that bounds its claim. The component-rule comparison (13 versus 1 false causal claims out of 34 noncausal cases) quantifies why the discipline matters.

Real-world applications (as implications of the protocol, not claims tested in the paper):

• Auditing semantic parsers and natural-language interfaces to databases, where a parser that handles novel recombinations must be shown to track roles rather than surface cues.
• Robustness reviews for compositional command understanding in deployed language systems, where held-out recombination accuracy is already a standard check.
• Documentation and model cards that must state not only benchmark accuracy but which representational explanation of that accuracy is licensed, and at what scope.
• Internal interpretability review of intervention-based claims, replacing probe accuracy and patching effects with a fixed evidence-to-claim mapping.

Industry relevance. Teams that use probing and activation patching to justify design decisions get an explicit rule set for when such evidence supports a causal claim, plus a released codebase and testbed that make the procedure reusable.

Future Directions

• Other architectures and large language models. All audits use T5 models; the authors state the verdicts have not been tested on other architectures or on large language models.
• Naturalistic data. The testbed is synthetic and template-clean, which enables exact interventions but leaves naturalistic language untested; transfer beyond the generator is shown only on one prespecified SLOG subdomain, itself synthetic.
• Validated detectors and a broader closure basis. Detectors remain unvalidated in trained models and only the cross-step class has an engineered known positive; the excluded object classes (nonlinear manifolds, higher-order or polysemantic interactions, sparse-feature circuits, diffuse long-range dynamic routes, nonlocal multi-source mixtures) are open territory that later work could add.
• Sharper Present-only boundaries and lower onboarding cost. Both rules are weakest at the Present-only boundary, and each new family must supply and validate at least six mechanism-specific interfaces plus a closure basis for a family-level negative.

Target Audience

Researchers and graduate students in NLP interpretability, compositional generalization, and representation learning who need to make or evaluate representation-level claims; benchmark designers who want to report which explanations of systematicity are licensed alongside accuracy; and applied scientists and reviewers who use probing and intervention methods and want explicit criteria for when those methods support causal conclusions. The paper is not beginner-oriented—it presumes comfort with probing, interchange interventions, and activation patching.

Authors’ abstract

Multiple studies have shown that neural networks can be behaviorally systematic, producing correct outputs for novel combinations of familiar parts. Whether they are also representationally systematic, that is, whether they represent these parts and their roles and use them to produce the outputs, remains hard to test, because each kind of evidence can mislead on its own: correct outputs can come from shortcuts, a decodable signal need not be used, an intervention can act through unintended pathways, and a failed search does not show absence. We propose a bounded evaluation protocol for representational systematicity (Bounded-RS): for an explanation that attributes a systematic behavior to a declared representation, it tests on a matched behavioral contrast whether that representation is recoverable and whether it is causally used, checks the validity of this evidence, and assigns the explanation one of four bounded states, Unsupported, Present-only, Causal-use, or Undetermined, through control-calibrated scores, gates, and stop rules. On a semantic-parsing testbed of 96 matched sets that we release, the tested role-filler Binding explanation of fine-tuned T5 models is Causal-use and the distributed Geometry explanation is Unsupported within its declared search space, verdicts that hold across calibration-compatible parameter configurations, training seeds, and T5-base. On 41 cases with known causal status, full adjudication licenses a false Causal-use in 1 of 34 noncausal cases, against 13 when recoverability is taken as use, without missing any causal case. Bounded-RS thus turns the question of representational systematicity, debated since Fodor and Pylyshyn, into comparable and falsifiable verdicts on declared explanations, each reported with the evidence pattern that licenses it and, for a negative, the search space that bounds it.