Masked Modeling and Predictive Pretraining

Masked modeling removes part of an observation. It asks the model to infer what is missing from context. In language, a token can often be predicted from surrounding syntax and semantics. In images, missing patches can be reconstructed from objects, texture, and spatial layout. In sensor streams, a hidden interval can be predicted from temporal patterns. The method forces the encoder to represent dependencies that make the hidden content predictable.

But the target matters, and two systems show how much. BEiT kept the corruption and changed only what was reconstructed. Pixels went out, discrete visual tokens from a learned tokenizer came in: “The pre-training objective is to recover the original visual tokens based on the corrupted image patches.” The payoff is in the same abstract — “base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K, significantly outperforming from-scratch DeiT training (81.8%) with the same setup”. An independent group, the masked-autoencoder authors, lists BEiT's ViT-B result at that same 83.2 and records that “BEiT reported a 1.8% degradation [2] when reconstructing pixels with ViT-B”. Same patches, same corruption, different target, 1.8 points.

data2vec moved the target off the input entirely. Published in 2022, it “predicts contextualized latent representations that contain information from the entire input” rather than words, pixels or visual tokens. Its authors then asked which internal layer should supply those representations. “Table 5 shows that the output of the feed-forward network (FFN) block works best while the output of the self-attention block does not yield a usable model.” On Librispeech, feed-forward-block targets give 13.1 WER. Self-attention-block targets give 100.0. Not a worse model — no model. Nothing changed between those two rows except which tensor was being predicted.

A powerful decoder can also absorb too much of the reconstruction burden, allowing the encoder to remain weak. Masking policy, mask ratio, target representation, and decoder capacity jointly define the learning problem. They should be selected with downstream use in mind rather than copied from another modality.

Changing any one of these can alter both compute and representation quality, and two of the four have been measured with everything else held fixed.

What to predict. I-JEPA ran that comparison with one line changed. With a ViT-L/16, computing the loss on target-encoder outputs gave 66.9% top-1 on the 1%-ImageNet-1K linear evaluation after 500 epochs. Computing the same loss in pixel space gave 40.7% after 800 epochs. The worse number is the one that cost more epochs. “From Table 7, it is clear that predicting in pixel-space leads to a significant degradation in the linear probing performance.”

Where to reconstruct. SimMIM swept the prediction head on a Swin-B backbone, from a single linear layer up to an inverse Swin-B. Heavier heads reconstructed better and transferred worse: “While generally heavier heads produce slightly lower losses, for example, 0.3722 (inverse Swin-B) versus 0.3743 (a linear layer), the transferring performances on the down-stream ImageNet-1K task are lower. It indicates that stronger inpainting capability does not necessarily result in better down-stream performance.” The inverse Swin-B head carries 174.8M parameters and 2.3x the training cost, and fine-tunes to 82.5% top-1 against the linear layer's 82.8%. The authors' own summary of the sweep: “the prediction head can be as light as a linear layer, with no worse performance than heavier ones”.

The same headline idea produces different engineering and semantic pressures. The temporal column is the one most often asserted rather than shown, so it is worth naming a system. wav2vec 2.0 arrived in 2020. It masks the speech input in latent space rather than the raw waveform, and fine-tuned on all of Librispeech it reaches 1.8/3.3 WER on test-clean/test-other. The result that shows what the pretraining actually bought is the low-label one: “Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER.” Ten minutes of transcription set against 53,000 hours of unlabelled audio. That is the argument for temporal masking, in a form a reader can go and check.

The optimal difficulty is empirical and modality dependent. Each of these failure modes has been measured somewhere, and the measurement is what tells you which direction you are failing in.

Ablations reveal which part of the objective creates transfer value, and step 4 is the one whose result is easiest to miss. The masked-autoencoder paper ran exactly that sweep: a ViT-L/16 on ImageNet-1K, decoder depths of 1, 2, 4, 8 and 12 Transformer blocks. Linear probing reads 65.5%, 70.0%, 71.9%, 73.5% and 73.3%. Fine-tuning over the same five runs moves only between 84.8%, 84.9%, 84.9%, 84.9% and 84.4%. The authors' summary is one sentence: “A sufficiently deep decoder is important for linear probing.” An 8-point swing under one protocol is invisible under the other. A sweep reported through a single evaluation can conclude that the component does not matter. Step 5 has a concrete case behind it too: wav2vec 2.0 shows the transfer value at its largest with ten minutes of labelled data, not with the full corpus.

Corruption design controls whether the model learns local repair, global semantics, or temporal structure. At a fixed backbone and budget, the pattern alone can outweigh most other choices. I-JEPA held the model and the schedule constant — ViT-B/16, 300 epochs, linear evaluation on 1% of ImageNet-1K — and varied only the masking strategy. Multi-block masking reached 54.2% top-1. Single-block masking reached 20.2%, random masking 17.6%, rasterized quadrant masking 15.5%. The caption of Table 6 states the conclusion plainly: “The proposed multi-block masking strategy is helpful for guiding I-JEPA to learn semantic representations.” One caveat belongs with the figures. The context budget is not identical across the four rows: Table 6 lists an average context ratio of 0.25 for multi-block and rasterized masking and 0.40 for block and random masking. So this compares masking strategies as configured, not patterns at a fixed number of visible patches. Even with that qualification, the spread from 15.5% to 54.2% comes from the corruption policy and nothing else in the architecture.