Contextual Representations and Transformer Encoders

The token “charge” turns up in all five sentences below. Each time it carries a different sense, and sometimes a different grammatical role.

A static embedding looks up one stored vector for a word type. A contextual encoder does something else: it combines the token with the tokens around it and produces a representation for that occurrence, at each layer.

The resulting vector can reflect sense, syntax, reference and task-relevant context. It remains learned evidence, not a transparent symbolic interpretation. Different layers and different pooling choices emphasize different properties.

A contextual vector is a function of the sentence, not a row in a table. Peters and colleagues put that difference into one clause in 2018. Their word vectors “are learned functions of the internal states of a deep bidirectional language model (biLM)”, one “pre-trained on a large text corpus”. Added to existing models, they report, the vectors “significantly improve the state of the art across six challenging NLP problems”. The six include “question answering, textual entailment and sentiment analysis”. Depth is doing work here, not decoration. They found that “exposing the deep internals of the pre-trained network is crucial”, because it allows “downstream models to mix different types of semi-supervision signals”. The layer is part of the representation.

Which layer carries what has been measured. In 2019 Tenney and colleagues put scalar mixing weights over the 24 layers of BERT-large and asked where each linguistic task lives. The order they found is the order a textbook would give: “We observe a consistent trend across both of our metrics, with the tasks encoded in a natural progression: POS tags processed earliest, followed by constituents, dependencies, semantic roles, and coreference.” But their two metrics do not agree on where the work happens, and the disagreement is the useful part. Mixing-weight mass concentrates in layers 9–20. The per-layer F1 gains are largest in layers 1–7. The authors attribute the gap to shallow shortcuts resolving the easy cases early. A single number like “layer 12” is not a property of the model. It is the answer to one particular question asked of it.

The architecture details belong elsewhere. This view is about the NLP interface. Subword IDs, positions, separators and masks define the sequence. Each encoder layer combines information across the visible positions. Every position then holds a vector conditioned on the sentence. From those, a task representation is formed — by selecting tokens, pooling a span, taking a special position, or aggregating a document. The result is fine-tuned, probed, retrieved against, or fed to a task head.

Two decisions in that chain do most of the damage or most of the good: which layer, and which pooling rule. They are made at stages three and four. Both are measurable.

Pooling is a modeling decision. It can hide or preserve different information, and the size of that effect has been measured on the two rules practitioners reach for first.

Both were measured in 2019, in the Sentence-BERT paper. Reimers and Gurevych open by naming the default and then dismantling it: “The most commonly used approach is to average the BERT output layer (known as BERT embeddings) or by using the output of the first token (the [CLS] token). As we will show, this common practice yields rather bad sentence embeddings, often worse than averaging GloVe embeddings (Pennington et al., 2014).” Their Table 1 gives the scale of it. Averaged across seven STS datasets, averaging BERT’s output layer scores a mean Spearman correlation of 54.81. The [CLS] output scores 29.19. Averaging static GloVe vectors — a lookup table with no context in it at all — scores 61.32. SBERT-NLI-base, trained for the job, reaches 74.89. The pooling rule most tutorials teach for classification is, on this task, less than half as good as the context-free baseline it was supposed to have replaced. A separate group reproduced the finding independently in 2020: BERT sentence embeddings “even underperform the GloVe (Pennington et al., 2014) embeddings which are not contextualized”.

Cost separates the same way. Finding the most similar pair in a collection of 10,000 sentences takes about 50 million inference computations with a BERT cross-encoder, roughly 65 hours. With SBERT sentence embeddings it takes about 5 seconds. One representation choice, four orders of magnitude.

Read the four options with those numbers in view. Special-position pooling uses a designated sequence token, or its transformed output. It is simple and fixed in size, and common for classification. It also depends on what pretraining and fine-tuning did to that position, is liable to underrepresent long documents, and, untuned, is the 29.19 above. Mean or max pooling aggregates the visible token vectors: length-insensitive in size, a simple baseline, able to blur rare evidence, and correct only if padding is masked properly. Span pooling combines states over an entity or phrase boundary. It supports extraction and relations, needs reliable spans, can carry boundary features, and preserves local evidence. Hierarchical aggregation encodes chunks and then combines the chunk representations. It handles long documents, adds a second model layer, needs cross-chunk tests, and suits sections and conversations.

None of the four is right by default. Each is a hypothesis about where the evidence sits. The STS table shows what an untested hypothesis costs.

Do not choose by parameter count or a generic leaderboard. Five checks, two of them with published numbers attached.

1. Match language and domain — tokenizer efficiency, pretraining coverage, licenses, domain vocabulary. Continuing BERT’s pretraining on PubMed abstracts and PMC full-text articles bought something measurable, and how much depends entirely on the task. The BioBERT paper reports it: “While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement).” One corpus change, three tasks, a payoff running from 0.62 to 12.24. Read the baseline before reusing those figures. They are improvements over the previous state-of-the-art models, not over general-domain BERT. BERT itself already sat 7.0 MRR above the prior state of the art on question answering, and 2.01 F1 below it on named entity recognition. BioBERT’s margin over BERT is therefore narrower than the headline number on QA and wider on NER. Measure domain fit on your own task, because the paper’s own three tasks disagree by a factor of twenty. Microsoft Research, working on the same question, reports that its PubMedBERT, pretrained from scratch on in-domain text, “is able to obtain consistent gains over BioBERT in most tasks”, and that “mixing out-domain text in pretraining generally leads to worse performance”.

2. Match sequence needs — measure truncation, chunking, and the cross-document context the task actually requires. Do not assume the default window covers it.

3. Define representation extraction — layer, pooling, span handling, task head. The layer cannot be copied out of a paper, because the papers disagree. Rogers and colleagues surveyed over 150 BERT analysis papers in 2020 and found one settled result: “There is a wide consensus in studies with different tasks, datasets, and methodologies that syntactic information is most prominent in the middle layers of BERT.” The supporting numbers are specific. Hewitt and Manning best reconstruct tree depth at layers 6–9 for BERT-base and 14–19 for BERT-large. Goldberg finds subject-verb agreement peaking around layers 8–9. And “The final layers of BERT are the most task-specific”. Outside that consensus the rankings contradict one another. Tenney’s pipeline ordering for chunking is reversed by Liu and colleagues, who find POS tagging and chunking best at the middle layers in both BERT-base and BERT-large. Two published answers, one model, opposite advice. Which is why the layer is something you measure on your own task.

4. Compare adaptation regimes — frozen features, probing, full fine-tuning, and smaller baselines, on the same split and the same budget.

5. Audit behavior and cost — slices, shortcuts, calibration, latency, memory, and the burden of updating the model later.

A probe may show that tense, syntax or entity type is recoverable from a hidden state. That finding does not prove the production classifier relies on the feature, nor that the representation mirrors a linguistic theory.

Use interventions, ablations, counterfactual inputs, attention-independent tests and downstream error analysis before making stronger claims.

There is a control for exactly this, and it is simple. Beside each linguistic task, put a control task that associates “word types with random outputs”. Hewitt and Liang proposed it in 2019, noting that “By construction, these tasks can only be learned by the probe itself”. A probe’s selectivity is the gap between the two accuracies. The result is uncomfortable. They “show that popular probes on ELMo representations are not selective”. Dropout, the usual lever on probe complexity, “is ineffective for improving selectivity of MLPs”. And probes on ELMo’s first layer give slightly better part-of-speech accuracy than the second, while probes on the second layer are “substantially more selective”. Two opposite layer rankings, one representation, depending on which question was asked.

The counterfactual input has a worked example too. McCoy and colleagues built HANS from templates in 2019 — 1,000 examples per template, 10,000 per heuristic, across lexical overlap, subsequence and constituent — so that three plausible shortcuts all give the wrong answer. Models that score well on MNLI, BERT included, fell to accuracies below 10% in most cases. Chance is 50%. The caption to their Figure 1 explains why the failure is that lopsided: “That is, they nearly always predicted entailment for the examples in HANS, leading to near-perfect accuracy when the true label is entailment, and near-zero accuracy when the true label is non-entailment.” The repair was small. Retraining on MNLI augmented with 30,000 HANS-style examples — about 8% of MNLI’s 392,702 training examples — largely fixed it. A held-out score from the training distribution could not have found any of this.

The intervention has one too. Replace the probe with a causal edit: remove a property from BERT’s representation, then measure what happens to the model’s own behaviour. Elazar and colleagues did that in 2021, and decodability and use came apart. “Our findings demonstrate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.” Not weakly correlated. Not correlated.

Use one encoder with special-position, mean and hierarchical pooling on a long-document classification task. Keep the split, the tokenizer and the classifier budget fixed.

Report truncation, class metrics, latency, and the examples where evidence appears late or in one short span. Then explain which pooling rule aligns with the task’s evidence distribution.

Expect the answer to be a property of your corpus rather than of the model. That is how it came out when the question was run at scale. Longformer, published in 2020, replaced full self-attention with a linear-scaling local-plus-global pattern. Why that matters, in its authors’ words: “This is an advantage for natural language tasks such as long document classification, question answering (QA), and coreference resolution, where existing approaches partition or shorten the long context into smaller sequences that fall within the typical 512 token limit of BERT-style pretrained models.” Their own results then show the gain tracking document length, not architecture. Against RoBERTa-base, Longformer-base gains 7.4 F1 on Hyperpartisan news detection, moving 87.4 to 94.8; mean context there is 705 wordpieces, 95th percentile 1,975. On IMDB, mean context 300 wordpieces, it gains 0.4 accuracy: 95.3 to 95.7. And on WikiHop the same model drops from 73.8 to 71.7 when it is configured at sequence length 512 with full attention. The long-document machinery pays where the evidence is long. Where it is not, it is nearly free of effect.

A different group named the same limitation the same year, building Big Bird — “one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length” — and reported sparse attention that “can handle sequences of length up to 8x of what was previously possible using similar hardware”. Your experiment is the small version of theirs. Measure where the evidence lives before choosing what reads it.