Intervenable Concept Dynamics for Video Activity Forecasting

AI summary

Overview

• Research area: Computer Vision — video activity recognition and anticipation, combined with interpretable machine learning (concept bottleneck models).
• Technical level: Advanced.
• Scope: The paper introduces TRACE-CBM, an autoregressive concept bottleneck model that forecasts future video activities by recursively evolving a named, human-editable concept state over sparse spatio-temporal graphs rather than rolling out opaque latent features.

What This Paper Is About

Concept bottleneck models (CBMs) force a prediction to pass through human-understandable concepts, which makes the intermediate state inspectable and editable. Standard CBMs, however, only ground concepts in the observed input: they do not describe how concepts should change over time. Video activity forecasting needs exactly that, because a prediction several steps ahead depends on a future semantic state, not just the current one.

The paper asks whether that future state can stay in concept space — named, readable and editable — instead of being pushed into an opaque latent trajectory. Achieving this would let a user correct or steer a forecast at inference time without retraining the model, and would let the effect of an edit be traced forward through the prediction horizon.

Key Contributions

1. TRACE (Temporal Relational Activity Concept Evolution): an autoregressive CBM for video forecasting that predicts human-readable concept states using sparse spatio-temporal graphs and activity-conditioned transitions.
2. Persistent concept-space forecasting: because forecast states stay in the same concept space as observed states, interventions on concept nodes, activity beliefs and learned relations propagate through the rollout to future time steps rather than only affecting the current prediction.
3. An intervention interface without retraining: three edit types are supported at inference — replacing a named concept activation in [0, 1], modifying an activity distribution, and modifying a learned edge — all decoded by the same shared activity head.
4. Experimental quantification of propagation: the paper measures how concept-node edits, activity-belief edits and edge sign flips change forecast correctness, and tests whether a vision-language model (Qwen3.6-35B-A3B) can act as an external agent that plans edits.

Main Findings

• Observed-activity accuracy: TRACE has the highest activity Top-1 accuracy among concept-based methods on all three datasets (BARISTA 76.0, Breakfast 45.5, MPII Cooking 2 Attr 47.0, over five seeds). The abstract reports this as a gain of 1.2–8.5 percentage points over concept-based methods.
• Forecast accuracy gap to black boxes: at horizon H3 (H = 3), TRACE reaches Top-1 of 67.6 (BARISTA), 37.5 (Breakfast) and 38.8 (MPII), which the abstract describes as within 1.9–2.2 points of matched black-box forecasters (feature transformer: 69.6, 39.7, 40.7). The Sparse Linear concept baseline scores 68.5, 43.4 and 37.7 on the same H3 metric.
• Ablation: removing score normalization hurts most consistently (BARISTA 64.3, Breakfast 39.6, MPII 35.0 versus the full model's 67.4, 42.9, 38.7). Replacing the graph with a dense rollout improves BARISTA (70.2) but costs 2.7 and 2.8 points on Breakfast and MPII. The paper reports the full model at the highest mean H3 Top-1 across the three datasets (49.6%).
• Sparsity of the graph matters: the dense rollout has 87 effective H3 parents versus 5 for the graph rollout, and the top-5 concepts carry 15% of the influence under the dense rollout versus 75% under the graph rollout.
• Concept-node edits repair forecasts: with a budget of 5 cumulative edits, concept-node edits correct 25.6% (BARISTA), 25.6% (Breakfast s1) and 34.4% (MPII) of initially incorrect H3 forecasts, with a C→W harm rate of 0.0 on all three. True-class probability rises by 7.2, 11.3 and 9.8 points respectively.
• Activity-belief edits are stronger where labels are informative: with b = 3, they correct 12.2% (BARISTA), 61.1% (Breakfast) and 46.7% (MPII) of wrong H3 cases, with harm rates of 0.0, 4.4 and 2.2.
• Edge edits barely matter: flipping the strongest learned relation sign changes true-class probability by −0.02, −0.00 and +0.04 points, which the authors read as robustness to individual relational perturbations and as evidence that edges are mainly for inspection.
• VLM-planned interventions: blind Qwen lowers true-class probability (−2.9, −13.0, −9.0), but oracle-assisted one-shot planning (+12.9, +23.1, +20.6) and sequential planning (+16.6, +32.4, +25.7) are effective, with sequential planning approaching the greedy oracle (+17.8, +34.4, +28.5).
• Steering to a requested target: the target heuristic reaches the requested H3 activity in 90.0% of Breakfast cases, 84.4% of MPII cases and 3.9% of BARISTA cases at Top-1; Qwen steering reaches 88.9%, 78.3% and 10.6%. Both beat the random baseline.

Methodology in Plain English

The pipeline has four stages.

1. Turn video windows into concept scores. Each video is split into non-overlapping windows. A frozen vision-language backbone (PE-L14) embeds both the windows and a vocabulary of 128 short textual concepts, and cosine similarity gives a raw score per concept per window. Concepts are proposed offline by a VLM (Qwen3.5-27B), then retrieved, visually verified, deduplicated and filtered for label overlap.
2. Normalize into bounded concept states. Because similarity scores are not comparable across concepts, each concept gets its own learned threshold and positive scaling factor, passed through a sigmoid to land in (0, 1). A validity mask keeps padding from affecting any computation or loss.
3. Refine and roll out on a graph. Three graph modules — a shared refinement module, an activity module and a forecast module — pass messages between concepts. Each module uses three relation families: same-time relations between different concepts, self-temporal relations carrying a concept to itself at the next window, and cross-temporal relations carrying one concept to a different concept. Only adjacent windows

Authors’ abstract

Concept bottleneck models route predictions through human-understandable concepts, making their intermediate states inspectable and editable. Extending this model family to forecasting requires modeling how concept activations evolve over time as objects, actions, and their interactions change, yet existing CBMs ground concepts only in observed inputs and do not forecast their future evolution. We study this problem using videos of ongoing activities and forecast future activities through a named-concept state. We propose TRACE-CBM (Temporal Relational Activity Concept Evolution), which first refines observed concept activations with sparse spatio-temporal graphs. It then recursively predicts future concept states, conditioning each transition on the prior activity distribution, and decodes both observed and predicted states with a shared activity classifier. At inference time, the model supports editing concept values, activity activations, and learned edges without parameter updates, exposing the relational spatio-temporal dependencies used for forecasting. Across multiple datasets, TRACE improves observed-activity Top-1 accuracy over concept-based methods by 1.2-8.5 percentage points and attains H3 forecast Top-1 accuracy within 1.9-2.2 points of matched black-box forecasters. Intervention experiments show concept-node edits correct 25.6-34.4% of initially incorrect H3 forecasts without flipping initially correct ones, and concept steering achieves the requested target activity in up to 90.0% of cases.