Rethinking Video Storage for AI Training: Breaking Cross-Frame Dependencies for Efficient Random Access

AI summary

Overview

Research area: Video data storage and loading systems for large-scale AI training (computer vision, video coding, data pipelines, and robotics/vision-language model training).

Technical level: Advanced. The paper assumes familiarity with inter-frame prediction, I/P/B pictures, GOPs, and decoded picture buffers, alongside ML data-loading concepts.

Scope: The paper proposes co-designing an online dependency-aware sparse decoding schedule with an offline "Dependency-Decoupled Random Access" (DDRA) encoding structure to reduce wasted decoding work for randomly sampled video frames, evaluated on a three-dataset robotics corpus with downstream VLA, WAM, and VLM training checks.

What This Paper Is About

Video datasets for AI training are stored using codecs designed for sequential playback, but training workloads instead sample short windows or sparse frames at random from long recordings. Because conventional codecs chain frames together through inter-frame references, serving one requested frame can force the decoder to reconstruct many frames nobody asked for, wasting CPU time.

The paper names these two wastes "decode amplification": sequential reconstruction of unrelated frames, and reconstruction of the transitive reference chains a sampled frame depends on. The goal is to cut both while keeping inter-frame compression (rather than falling back to independent JPEG frames or very short GOPs, which reduce decoding work but inflate storage and transfer volume).

Key Contributions

1. A decomposition of decode amplification. The authors formalize decode amplification as a ratio of reconstructed pictures to requested pictures, and split it into an "unrelated reconstruction" term (frames outside the dependency closure) and a "reference reconstruction" term (frames inside the closure but not requested).

2. Dependency-aware sparse decoding (online, no transcoding). The loader parses headers to build a reference graph, computes the dependency closure of the request by breadth-first traversal, then walks the segment in decoding order — fully reconstructing only frames in the closure, running a state-only path for skipped frames, and converting only requested frames to RGB.

3. Dependency-Decoupled Random Access (DDRA) encoding (optional, offline). DDRA removes B-to-B reference edges and restricts each interior B frame to the two intra-coded anchors bounding its intra period. The resulting dependency closure is bounded at at most three pictures per requested target, independent of the intra period, while retaining inter-frame prediction and standards-compliant bitstreams.

4. An evaluation spanning loading efficiency, storage cost, and downstream training. The study covers three robotics datasets (3,000 episodes, 8,908 camera streams), four access patterns, bandwidth- and resource-sensitivity sweeps, an intra-period sweep, a component ablation, and VLA/WAM/VLM training checks under compression.

Main Findings

• Sparse decoding alone beats existing loaders without changing stored video. Averaged across four access patterns, the native sparse loader is 1.81×, 1.34×, and 1.25× faster than PyAV, Decord, and TorchCodec respectively.

• DDRA transcoding plus sparse decoding gives the best throughput-payload trade-off. The H.265 variant reaches 1.45× the throughput of LeRobot v3 with 58.7% less media payload, and 2.02× the throughput of JPEG with 90.0% less payload, at comparable reconstruction quality (PSNR within 0.18 dB of the LeRobot reference). Across the H.264/H.265 pair the ranges are 1.42–1.45× LeRobot v3 throughput with 29.8–58.7% less payload, and 1.99–2.02× JPEG throughput with 83.0–90.0% less payload.

• Random-access latency drops sharply. In random access mode, average loading time falls from 297.8 ms/batch (TorchCodec) to 134.1 ms (DDRA H.264) and 140.6 ms (DDRA H.265) — 2.22× and 2.12× respectively. Both DDRA variants beat LeRobot v3 and JPEG in mean and p95 loading time in every main-table pattern.

• Payload numbers are concrete. Media payload is 14.62 MiB/episode for the native bitstreams, 121.54 for JPEG, 29.45 for LeRobot v3, 20.68 for DDRA H.264, and 12.15 for DDRA H.265.

• Amplification analysis quantifies the two effects. With K=10, raising stride from 1 to 30 lifts Sequential's expected amplification from 2.80 to 16.40; at stride 30, Sparse gives 5.04 and DDRA gives 2.01. At d=4, raising K from 1 to 30 lowers DDRA from 2.94 to 1.16. At K=10 and d=4, raising the intra period from 4 to 32 lowers DDRA from 1.83 to 1.28, while Sparse moves from 2.33 to 2.63.

• The two components contribute separately. Under random access, sparse decoding cuts reconstructed frames from 233.9 to 92.3 per episode and loading time from 668.3 to 297.8 ms (2.24×); adding DDRA takes these to 43.1 frames and 134.1 ms (a further 2.22×). Combined speedups over Sequential are 2.73–4.98× across patterns.

• Advantages hold under bandwidth caps. At a node-wide cap of 100 MiB/s, DDRA H.264 and H.265 sustain 6.38 and 10.79 ep/s, versus 4.36 ep/s for LeRobot v3 and 1.06 ep/s for JPEG. At 800 MiB/s, DDRA H.264 exceeds the native sparse path, consistent with decoding efficiency mattering more once transfer constraints relax.

• Gains are robust across configurations. DDRA's loading advantage persists across all 27 tested access/resource configurations, with speedups over the native sparse path of 1.44–4.41× for H.264 and 1.59–3.27× for H.265.

• The intra period is a genuine trade-off. DDRA H.265 (stride 4) has its smallest payload at T_I = 8 and lowest average loading time at T_I = 32; moving from 8 to 32 lowers loading time by 14.8% while raising payload by 4.7%.

• Sparse decoding is bit-exact. Across 100 streams and 10,000 requests of ten frames each (100,000 returned frames), outputs were pixel-identical to full sequential FFmpeg RGB24 decoding, with maximum absolute RGB error of zero.

• Compression does not measurably hurt downstream tasks in the tested settings. DDRA H.265 requires roughly 50% less storage and transfer bandwidth than LeRobot v3 while maintaining comparable VLA and WAM performance; at approximately 100× compression relative to the Original representation, the authors report no statistically significant success-rate change (p > 0.05) for either VLA or WAM. For VLM, transcoding did not necessarily reduce accuracy when video quality stayed high.

• The authors state their own limits. The conclusion notes that the preferred representation depends on request span and bandwidth, and that the reported loading rates do not establish full-tensor delivery rates or training-time speedups.

Methodology in Plain English

The work starts by observing that a training sampler asks for a handful of frames, but the decoder has to produce those frames through two extra costs: everything it decodes on the way (unrelated frames) and everything the requested frame references (dependency chains).

To fix the first cost, the loader first reads the video headers without decoding pixels, building a map of which frame references which. Given a request, it walks that map backward from the requested frames to find the minimal set that must be reconstructed. It then plays through the file in decoding order, but only actually reconstructs pictures in that minimal set; for everything else it advances the decoder's internal bookkeeping (frame ordering and reference-buffer state) without doing pixel reconstruction. Only the requested frames are converted to RGB and returned in the original request order, duplicates included.

To fix the second cost, the authors change the encoding itself. Instead of a hierarchical pyramid where B frames reference other B frames, DDRA makes every interior frame reference only the two intra-coded anchor frames that bracket it. A requested interior frame therefore needs at most itself plus two anchors. The implementation uses modified libx264/libx265 builds with an extended B-frame limit (the custom x264 build raises the limit to 32), a fixed keyframe interval of 32, open GOPs, and ref=1.

The evaluation stores every representation as episode-level Lance records on object storage, reads them through the same path on one 16-CPU, 64-GiB node with 32 workers and batch size eight, times 500 measured batches per method–pattern after ten warm-up batches, and averages per dataset before averaging across datasets so that the reported p95 is not a pooled percentile. Reconstruction quality is calibrated so all transcoded representations sit within 0.18 dB of the LeRobot reference. Downstream, they fine-tune π0.5 on LIBERO for VLA, fine-tune OpenWAM for WAM, and use Qwen3.5-2B on MVBench for VLM across compression sweeps.

Why This Matters

For research: The paper reframes video storage as a workload-aware problem rather than a playback-quality problem, and gives a measurable decomposition (unrelated versus reference reconstruction) that other storage and loader designs can be scored against. It shows a middle path between playback-oriented codecs and image-per-frame storage without giving up inter-frame prediction.

Real-world applications:

• Robot learning pipelines: the evaluation corpus is robotics data (RealSourceWorld, RoboCasa, Table30v2), where policies are trained on sparse frame windows across thousands of camera streams.
• Autonomous driving datasets: long multi-camera recordings that are repeatedly subsampled across epochs, where both storage volume and per-sample decode latency are binding constraints.
• Video-language and multimodal training: the VLM results suggest that quality-aligned transcoding can preserve accuracy while shrinking transfer footprint.
• Training clusters on object storage: the bandwidth-capped experiments map directly onto cloud/OSS-backed training where network capacity, not CPU, is the limiter.

Industry relevance: The gains apply without changing the decoded-frame interface that existing training pipelines expect, and the sparse-decoding half requires no re-encoding of archives. That makes it deployable as a loader upgrade on existing video corpora, with DDRA transcoding as an optional follow-on for teams willing to re-encode. The measured payload reductions (58.7% versus LeRobot v3, 90.0% versus JPEG) translate directly into storage and egress cost.

Future Directions

• Extending DDRA beyond H.264/H.265. The paper's implementation depends on custom encoder builds (a modified x264 with a raised B-frame limit); porting the bounded-reference structure to other codecs and to hardware decoders is an open engineering question.

• Recovering compression lost to the bounded reference structure. Removing B-to-B edges and limiting each interior frame to two anchors trades coding efficiency for bounded reconstruction; better prediction under that constraint, or joint rate–access optimization, is a natural next step.

• Closing the gap to end-to-end training speed. The authors explicitly note that the reported loading rates do not establish full-tensor delivery rates or training-time speedups; measuring the full data path through GPU transfer and model computation remains to be done.

• Request-adaptive representation selection. The results show the best choice depends on request span, bandwidth cap, and intra period (payload minimized at T_I = 8, loading time at T_I = 32), which motivates policies that pick a representation or intra period per workload rather than per corpus.

Target Audience

ML systems and data-infrastructure engineers building training pipelines over large video corpora; video coding researchers interested in random access and reference-structure design; robotics and autonomous-driving teams training policies or world models on multi-camera recordings; and graduate students studying the intersection of codec design and ML data loading. Readers without background in inter-frame prediction will find the motivation and results accessible, but the method sections require codec familiarity.

Authors’ abstract

Large-scale AI training increasingly relies on massive video datasets with random temporal sampling patterns. However, existing video storage paradigms are optimized for sequential playback and employ sophisticated inter-frame reference structures to maximize compression efficiency. When applied to the random-sampling workloads of AI training, these paradigms face substantial decoding-efficiency bottlenecks. Workarounds such as independent JPEG frames and short groups of pictures (GOPs) reduce decoding work at the cost of larger storage and transfer volumes. In this paper, we identify two sources of decode amplification: sequential reconstruction of unrelated frames and reconstruction of the reference chains required by sampled targets. We address both sources by co-designing the decoding schedule and video reference structure. Dependency-aware sparse decoding reconstructs only requested frames and their required references, skipping unrelated reconstruction without transcoding the source videos. Optional offline Dependency-Decoupled Random Access (DDRA) encoding further reduces reference reconstruction by restricting each interior frame to two bounding intra-coded anchors. Together, sparse decoding and DDRA retain inter-frame prediction while bounding reconstruction to at most three frames per requested target, independent of the intra period. Experiments on a large-scale robotics corpus show that, at comparable reconstruction quality, our method achieves $1.45\times$ the loading throughput of LeRobot v3 with 58.7% less media payload.