Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understanding

AI summary

Overview

Research area: Computer vision — video action recognition, model editing, and fine-grained video understanding.

Technical level: Intermediate. The paper assumes familiarity with classification heads, weight vectors, vision-language models, and fine-tuning, but the core idea is explained geometrically (adding and subtracting weight vectors) and is accessible.

Scope: The paper defines a new task called category splitting, in which a trained video classifier is edited so that one coarse category becomes several fine-grained subcategories, and it proposes zero-shot and low-shot methods for doing so without retraining the backbone.

What This Paper Is About

Video recognition models are trained on fixed label sets that are often too coarse — a single label like "open" hides whether an object was opened by pushing, opened quickly, or only opened halfway. When new distinctions become important, the usual fix is to collect annotations and retrain, which is expensive. This paper asks how to split an existing coarse category into fine-grained subcategories by editing only the classifier, using no new video data or only one example per new subcategory, while leaving all other predictions unchanged.

Key Contributions

1. Defines the category splitting task. The paper formalises editing a classifier so that a coarse category c is removed from the label space and replaced by subcategories S^c = {s_1^c, ..., s_k^c}, giving a new label space 𝒴' = (𝒴 \ {c}) ∪ S^c, subject to two properties borrowed from NLP model editing: generality (correctly classify unseen examples of the new subcategories) and locality (preserve predictions on all other categories).

2. Proposes two zero-shot editing methods that require no video data. Modifier retrieval builds a dictionary of "modifier vectors" by grouping already-fine-grained categories in the classifier's head, subtracting the pseudo-coarse mean from each weight vector, and then retrieving and adding an appropriate modifier to the coarse category's weight. Modifier alignment trains a small MLP to map text embeddings directly into classifier weight space, so the method generalises to modifiers that never appeared in the original label set.

3. Shows low-shot fine-tuning is highly effective and benefits from zero-shot initialisation. Fine-tuning only the new subcategory weights — initialised either from the coarse weight or from the zero-shot modifier vector — with as little as one video per subcategory beats full-data fine-tuning on the combined generality/locality measure.

4. Introduces benchmarks and metrics for the task. Two new benchmarks, SSv2-Split (from Something-Something V2) and FineGym-Split (from FineGym288), each with two complementary subsets, plus generality and locality metrics. A project page is at https://kaitingliu.github.io/Category-Splitting/

Main Findings

• Zero-shot editing beats vision-language models by a wide margin on generality. On SSv2-Split subset A the method reaches 46.3% generality with 98.9% locality, versus 27.6% generality for CLIP, 30.9% for FG-CLIP and 28.6% for VideoCLIP-XL (all VLMs score 100.0 locality by construction since they do not touch the base model). On FineGym-Split subset A the method reaches 34.2% generality versus CLIP's 12.1% and the strongest VLM, VideoPrism, at 21.7%. On SSv2-Split subset B the method reaches 38.4% versus 30.8% for the best VLM.
• Video-text pretraining does not automatically help on SSv2-Split. VideoCLIP-XL, VideoPrism and InternVideo2 all perform worse than CLIP and FG-CLIP on subset B, and InternVideo2 is the weakest VLM on both subset A (25.9%) and subset B (21.8%).
• Mining modifiers from the classifier itself is what drives the gain. In the zero-shot ablation on SSv2-Split subset A, modifier retrieval lifts generality from the VLM baseline of 27.6% to 45.0% while keeping 98.9% locality; adding modifier alignment raises it a further 1.3 points to 46.3% with a mean of 72.6.
• Isolating the edit is essential. Updating the backbone, head and new head in the one-shot setting reaches 33.6% generality but destroys locality (0.0). Restricting updates to the head or to only the new head restores locality to 98.4 or higher while raising generality to 48.4% (coarse initialisation) or 45.0% (random initialisation).
• Zero-shot initialisation compounds with fine-tuning. Initialising the extended head from modifier alignment gives 52.8% generality and 98.2% locality (mean 75.5), which is +4.4 points of generality over coarse-category initialisation and +7.8 points over random initialisation. This one-shot configuration outperforms full-data fine-tuning in mean score (75.5 versus 54.4), because full-data fine-tuning biases the model to the new classes and collapses locality (22.1, with a large ±18.8 spread).
• Stronger video pretraining helps, but the method works on weak backbones. Editing a model trained from scratch still yields 37.0% generality and 97.7% locality (mean 67.4). Video-only pretraining outperforms CLIP's visual encoder (38.2%): VideoMAE 42.9%, MME 42.6%, SIGMA 44.1%, and the best result MVD at 46.3% (mean 72.6).
• The text encoder matters less than expected. A text-only encoder (RoBERTa) reaches 40.9% generality; CLIP reaches 46.3%; video-text encoders do not consistently beat it (InternVideo2 drops to 36.9%, VideoCLIP-XL 45.5%, VideoPrism 46.5%). The authors conclude multimodal alignment helps but image-text models are sufficient.
• Direction-type splits are easiest; object and interaction splits are hardest. Performance is highest for differences in direction, spatial position, motion and state change, and lowest for object count, action success/intent, and object interactions.
• Modifiers with existing analogues help, but are not required. Splits are easier when an analogous category sharing the same modifier already exists in the label space, yet the method still performs well without such analogues.
• Locality degrades slightly as splits get larger; generality does not. Increasing the number of subcategories in a split causes a small drop in locality with no clear trend in generality.
• The method tolerates less compositional structure in the original labels. When the fraction of coarse categories in the base label space rises from 50% to 66% and 75% (reducing fine-grained categories from 92 to 67 to 55), generality falls only from 46.3% to 44.2% and 44.6%, with locality stable at 98.8–99.1%, and the method still beats CLIP in every setting.
• Some new visual distinctions are captured, others are not. Because edits touch only the classification head, the backbone must already encode the needed cues. The method fails on concepts such as "continues" or "deflected", but succeeds on previously unseen concepts such as "breaks" or "slanted surface", which the authors attribute to the compositional structure of stronger pretrained backbones like MVD.
• Dataset scale. SSV2 contributes 220K videos over 174 fine-grained categories; FineGym288 contributes 31K videos over 288 categories. The constructed SSv2-Split has 54 coarse categories and FineGym-Split has 42, each split into 2–19 subcategories. Subset A of SSv2-Split uses 119 categories and 169K videos to train the base model, with 27 coarse categories split into 92 fine subcategories, evaluated on 429 generality and 24,347 locality videos.

Methodology in Plain English

The authors start from a classifier trained on a mixed-granularity label set — some labels coarse, some already fine. Each fine-grained label can be read as a coarse concept plus a modifier (for example, "pushing left to right" = "pushing" + "left to right"). Because the classifier already distinguishes several such variants, the differences between their weight vectors in the classification head already encode the modifier directions.

Modifier retrieval works in three steps. First, the authors group existing fine-grained categories that share a base concept into a pseudo coarse category and average their weight vectors to get a base vector. Second, they subtract that base vector from each fine-grained weight vector to get a modifier vector, storing each one in a dictionary keyed by its descriptive text. Third, to split a new coarse category, they describe the desired subcategory in text, match that description against the dictionary entries, and add the retrieved modifier vector to the coarse category's weight — creating a new classifier weight with no video data and no change to the backbone.

Modifier alignment extends this beyond the dictionary. A small MLP is trained to map text embeddings (from a frozen CLIP ViT-L/14 encoder) into classifier weight space. Its training pairs come from the modifier dictionary itself, plus category-text-to-weight pairs, so no video is needed. At inference, an unseen modifier text can be projected directly into a new weight vector. Only the MLP is trained; the classifier and text encoder stay frozen.

Low-shot splitting handles the case where one labelled example per subcategory exists. The coarse weight is removed and replaced with new subcategory weights, all initialised from the coarse weight (or, better, from the zero-shot modifier vector), and only these new weights are fine-tuned with cross-entropy. Freezing everything else is what keeps locality high — updating the backbone in this regime drives locality to 0.0.

Both benchmarks are built by collapsing groups of original fine categories into coarse ones, then training base models on mixed-granularity subsets A and B so that every category is evaluated as a split target in one of the two subsets. The base model is a ViT-Small pretrained with MVD on Kinetics-400, fine-tuned using MVD's SSV2 recipe on 4 NVIDIA A100 GPUs with batch size 18; low-shot training uses AdamW with learning rate 1×10⁻³, weight decay 1×10⁻³ and batch size 16.

Why This Matters

Impact on research. The paper reframes fine-grained video recognition as a model editing problem rather than a data-collection problem, and shows that compositional structure already latent in a trained classifier head can be exploited to expand the label space. It also provides the first benchmarks and metrics (generality and locality) for evaluating this behaviour in video, and it shows that large vision-language models are not automatically the right tool for fine-grained temporal distinctions — a result with implications for how the community builds fine-grained recognisers.

Real-world applications:

• Adapting deployed video classifiers when a taxonomy is refined — for example, distinguishing ways an object was opened, or severities of an action — without retraining or re-annotating.
• Rare or anomalous events in specialised domains (surveillance, safety monitoring) where labelled examples for the new subcategory may not

Authors’ abstract

Video recognition models are typically trained on fixed taxonomies which are often too coarse, collapsing distinctions in object, manner or outcome under a single label. As tasks and definitions evolve, such models cannot accommodate emerging distinctions and collecting new annotations and retraining to accommodate such changes is costly. To address these challenges, we introduce category splitting, a new task where an existing classifier is edited to refine a coarse category into finer subcategories, while preserving accuracy elsewhere. We propose a zero-shot editing method that leverages the latent compositional structure of video classifiers to expose fine-grained distinctions without additional data. We further show that low-shot fine-tuning, while simple, is highly effective and benefits from our zero-shot initialization. Experiments on our new video benchmarks for category splitting demonstrate that our method substantially outperforms vision-language baselines, improving accuracy on the newly split categories without sacrificing performance on the rest. Project page: https://kaitingliu.github.io/Category-Splitting/.