Contrastive Learning for Authorship Verification

AI summary

Overview

• Research area: Natural Language Processing, specifically authorship analysis (authorship verification) and the fine-tuning of transformer language models.
• Technical level: Intermediate. The paper assumes familiarity with transformer architectures, bi-encoders versus cross-encoders, contrastive losses (InfoNCE / NT-Xent / Supervised Contrastive Loss), and metric-learning tooling such as PyTorch Metric Learning.
• Scope: A single-author study comparing contrastive learning against binary pair classification for transformer-based authorship verification, culminating in a ModernBERT bi-encoder that reports 98.4% accuracy on the PAN21 authorship verification task.

What This Paper Is About

Authorship verification asks whether two texts were written by the same person. Transformers can be fine-tuned either as cross-encoders (binary classification over a concatenated pair) or as bi-encoders trained with a contrastive objective, and it has been unclear which objective is actually responsible for strong performance once other factors are controlled. This paper holds pretrained model and context length fixed where possible, tunes batch size and learning rate separately for each objective, and compares the two approaches across five transformer architectures in the PAN21 fanfiction setting.

Key Contributions

1. A controlled comparison of training objectives. The paper compares contrastive (bi-encoder) learning against classification (cross-encoder) training for authorship verification, tuning batch size and learning rate separately for each approach while holding remaining training conditions fixed.
2. A systematic sweep of practical design factors. It examines loss function, batch size, training duration, pretrained model choice, input context length, pooling strategy, and random text span data augmentation in the PAN21 setting.
3. A state-of-the-art PAN21 result. A ModernBERT-large bi-encoder trained with a 4096-token context length reports 98.4% accuracy and 0.984 F1 on PAN21, with an overall score of 0.979 in the multi-metric comparison to prior work.
4. Released artefacts. Code and the dataset, in Parquet format, including the validation data split used, are released at https://github.com/petekirby/contrastive-av.

Main Findings

• Contrastive beats classification across architectures. At length 512 and 10 epochs, contrastive models outperformed classification models for every model tested: TinyBERT 0.812 accuracy / 0.812 F1 versus 0.745 / 0.761 for classification; DistilBERT-cased 0.877 / 0.874 versus 0.684 / 0.725; BERT-base-cased 0.891 / 0.881 versus 0.787 / 0.797; ModernBERT-base 0.900 / 0.898 versus 0.653 / 0.706; ModernBERT-large 0.913 / 0.911 versus 0.703 / 0.742.
• Contrastive wins even at half the context length. Contrastive results at 256-token context still exceeded classification at 512 tokens ("half-length test F1" of 0.789, 0.822, 0.836, 0.841 and 0.861 for the five contrastive models), while using less computation than 512-token classification.
• InfoNCE is the strongest loss tested. On TinyBERT at batch size 256, learning rate 2×10⁻⁵, 10 epochs, InfoNCE reached 0.753 validation accuracy / 0.774 F1, ahead of Circle (0.732 / 0.764), Multi-Similarity (0.714 / 0.748), and Semi-Hard Contrastive (0.704 / 0.734). SoftTriple and Proxy Anchor both collapsed to 0.500 accuracy / 0.667 F1.
• Proxy-based losses fail at this scale. SoftTriple and Proxy Anchor, which learn proxy embeddings per class, failed with 238,815 author classes in the training set.
• Mean pooling alone is best. With TinyBERT contrastive at batch size 1024, learning rate 4×10⁻⁴, temperature 0.01 and 40 epochs, mean-pooling only reached 0.861 validation accuracy / 0.863 F1, slightly beating a two-layer projection head with mean pooling (0.857 / 0.860), mean first–last layer pooling only (0.853 / 0.857), and a two-layer projection head with mean first–last layer pooling (0.851 / 0.856).
• Random text span augmentation is essential. Removing it dropped TinyBERT from 0.861 / 0.863 to 0.713 / 0.740 at 6 epochs, and further to 0.682 / 0.727 at 40 epochs, indicating overfitting.
• Longer context and larger batches improve results. For a ModernBERT-large bi-encoder at learning rate 1.2×10⁻⁴, temperature 0.01 and 40 epochs: context 512 / batch 1024 gave 0.926 accuracy and 0.925 F1; context 1024 / batch 1024 gave 0.952 / 0.955; context 2048 / batch 2048 gave 0.976 / 0.975; context 4096 / batch 4096 gave 0.984 / 0.984.
• New best PAN21 numbers. The ModernBERT bi-encoder scored AUC 0.995, c@1 0.982, F1 0.984, F0.5u 0.960, Brier 0.972, overall 0.979, ahead of the BigBird cross-encoder (0.990, 0.946, 0.944, 0.962, 0.956, 0.960) and Boenninghoff et al. (0.987, 0.950, 0.952, 0.938, 0.945, 0.955).
• Model size helps, but ModernBERT-base is an outlier under classification. ModernBERT-large classified better than ModernBERT-base, which the paper tentatively links to ModernBERT dropping the next sentence prediction task during pre-training.

Methodology in Plain English

The authors take a large fanfiction corpus in which each document is labelled with its author, then break the fixed same-author and different-author pairs into individual document rows. This lets them recombine pairs dynamically, producing billions of potential distinct negative pairs. Authors are sampled at random, two documents per author are chosen, and each document is rotated to begin at a random word boundary before truncation, so that a document yields many different truncated views. The paper describes this rotation as a data augmentation strategy that produces equal-length spans at any position with little disturbance to the data.

Two training setups are compared. In the classification setup, the two texts are concatenated into a single input sequence so the transformer can attend across them, and the output is a sigmoid score whose threshold is chosen to maximise F1 on validation data. In the contrastive setup, each text is encoded independently into one embedding, and InfoNCE loss with temperature scaling pulls same-author embeddings together and pushes different-author embeddings apart, using all other authors in the batch as negatives; cosine similarity is the scoring function and the decision threshold is again chosen to maximise validation F1.

Hyperparameters are tuned on TinyBERT and transferred to the larger models using a μTransfer-style approach; because Complete(d)P with α = 1 justifies no adjustment for depth, only the learning rate is adjusted for width, by a factor of 0.4 going from width 312 to 768 and 0.3 going from width 312 to 1024. Validation uses 10,000 same-author, different-fandom pairs and 10,000 different-author pairs, with authors and fandoms removed from training data. Model selection, hyperparameters and calibration use only the validation set. The final reported PAN21 model was trained for 40 epochs and uses Platt scaling with an abstention delta selected on the validation set, and its sigmoid-free similarity thresholding is based on cosine similarity.

Why This Matters

The paper's central claim is that for open-set authorship verification, learning a representation for each text independently and comparing embeddings afterwards is more effective — and computationally cheaper at a fixed context length — than letting a cross-encoder attend over both texts at once. The authors frame the contrastive objective as solving many implicit authorship attribution proxy tasks at once, treating InfoNCE as a categorical cross-entropy objective over one positive and many negatives. They note the computational advantage of delaying pairwise comparison until the cosine similarity step appears to outweigh any benefit of attention across every pair.

• Forensic and investigative text analysis: deciding whether a suspect's writing matches a questioned document, in the genre of the fanfiction data used here (documents up to 21,000 characters).
• Plagiarism and ghostwriting detection: flagging texts whose style does not match a claimed author's known corpus.
• Online identity linking and abuse investigation: matching accounts or messages to a single author across different platforms or fandoms, which is precisely the open-set, unseen-author, unseen-fandom condition that PAN21 tests.
• Verification of authorship claims in publishing or academia: supporting or challenging attribution of unsigned or disputed texts.

Industry relevance: the released model design is comparatively inexpensive — embeddings can be computed once per document and reused, rather than requiring a full cross-encoder pass for every candidate pair — which matters for large-scale deployment where many pairwise comparisons must be scored. The finding that a smaller context budget still outperforms a longer-context classification model suggests cost savings are available, and the paper's ablation results give practitioners concrete guidance on loss choice, pooling, batch size and augmentation.

Future Directions

1. Test the contrastive-versus-classification comparison beyond PAN21. The paper explicitly leaves it to future work to consider performance on authorship attribution, style change detection, and other datasets.
2. Determine whether longer context simply enables larger effective batches. The authors leave open whether a 1024-token context effectively means a larger batch size is supported by the additional data.
3. Explore the training objective further. Li et al. left the choice of training objective to future work, including whether contrastive learning improves over a standard classification objective — a question this paper addresses for PAN21 but does not resolve generally.
4. Explain the ModernBERT classification anomaly. The paper suggests, without confirming, that ModernBERT's weaker classification performance may relate to dropping next sentence prediction during pre-training.

Target Audience

Researchers and practitioners working on authorship analysis, stylometry, and text forensics; NLP engineers interested in contrastive learning, bi-encoder retrieval architectures, or PyTorch Metric Learning; and anyone evaluating design choices for fine-tuning transformers on paired-text verification tasks. Readers unfamiliar with InfoNCE or bi-encoder architectures will need background reading first, which places the paper at an intermediate level.

Authors’ abstract

Our results show that contrastive learning outperforms a classification-based approach to authorship verification under the tested settings. We identify loss function, batch size, training duration, pre-trained model, input context length, and random text span data augmentation as important factors of model performance. Based on these considerations, we develop a ModernBERT Bi-Encoder model that achieves 98.4% accuracy on the PAN21 authorship verification task.