Modeling Time-Lapse Trajectories to Characterize Cranberry Growth

AI summary

Overview

Research area: Computer vision for agriculture, specifically self-supervised representation learning and change monitoring for crop phenotyping (cranberry growth, ripening, and rot).

Technical level: Advanced. The paper assumes familiarity with vision transformers, self-supervised pretext tasks, contrastive/foundation-model feature extractors (DINOv2, SigLIP, Swin, ViT), UMAP dimensionality reduction, Bayesian Gaussian mixture models, and Grad-CAM.

Scope: The paper introduces a time-lapse trajectory (TLT) framework that fine-tunes frozen vision-transformer backbones with a small trainable encoder to produce interpretable 2D latent tracks of cranberry growth, and releases a 52-session time-lapse dataset of 8 cranberry varieties.

What This Paper Is About

Cranberry breeders and growers need to track how plants and fruit change across a growing season, but that monitoring is normally done by hand and requires expert time and judgment. Prior deep-learning approaches predict only a single change metric (such as rot or ripeness), which misses the combined complexity of crop development.

The goal of this work is to learn a single, interpretable latent space from time-lapse crop images in which the same region or berry traces a predictable path over time — a "trajectory" — that can be used both to forecast future crop state and to compare cranberry varieties without laborious image annotation.

Key Contributions

1. TLT framework: A method that models crop growth by learning latent representations from a time series of crop images captured at fixed spatial locations, using a frozen pre-trained feature extractor plus a jointly trained encoder and multiple pretext-task prediction heads.
2. TLT prediction module: A trajectory model that forecasts crop development through time-series observations, conditioned on variety-specific cranberry dynamics.
3. TLT analysis module: A module for breeders that provides an expected temporal track for a set of cultivars, revealing positive or negative deviations from desired phenotypes.
4. TLC (Time-lapse Cranberry) Dataset: A publicly released dataset imaging 8 cranberry varieties over one growing season (a span of around 4 months), annotated with fungicide treatment, fruit rot prevalence, and yields. Code and data are at https://github.com/ronan-39/tlt/.

Main Findings

• DINOv2 is the strongest single head for time prediction: With one time prediction head on image patches, DINOv2 reached an MAE of 3.39 ± 3.55 days, compared to ViT at 3.51 ± 3.17, SigLIP at 4.41 ± 4.58, and Swin at 5.166 ± 5.01, over a dataset with a total time span of 108 days.
• Adding a class head preserves time accuracy but splits the ranking: With time and class heads jointly, ViT led time prediction at MAE 3.49 days, while DINOv2 led class prediction at 79.4% percent agreement. Swin performed worst on class at 24.7%, marginally above random guessing (12.5%). ViT reached 63.1%, SigLIP 60.1%.
• More heads degrade class prediction: With time, class, and fungicide heads together, the top class result (DINOv2) fell to 51.4%. Time MAE was 5.65 for ViT, 5.76 for DINOv2, 5.74 for Swin and 5.70 for SigLIP. Fungicide prediction stayed strong across all backbones, from 77.5% (ViT) to 84.6% (DINOv2).
• Rot prediction on berries was robust to added heads: On bounded berry images with all four heads, DINOv2 achieved the best time (5.89 days), class (54.8%), fungicide (64.4%) and rot (94.8%) results. ViT scored 6.93 days, 45.9%, 62.7% and 93.5%; Swin 7.81 days, 46.1%, 63.7%, 93.5%; SigLIP 6.32 days, 45.2%, 58.3%, 94.7%. Adding the rot head did not degrade the other heads.
• The encoder generalizes to unseen genotypes: Using a modified percent agreement metric that assigns each Gaussian mixture component the class label it primarily contains, DINOv2 reached 91.8% on withheld classes A and B and 75.4% on D and G; SigLIP reached 82.8% on A, C, H and 61.3% on F, B, D. Time prediction on withheld genotypes stayed comparable to in-class prediction, with MAEs of 3.88 ± 3.06 (DINOv2, D, G), 4.70 ± 4.29 (DINOv2, A, G, H), 4.19 ± 4.17 (SigLIP, A, B) and 4.704 ± 4.29 (SigLIP, A, C, H).
• Latent space organizes by variety and time: Pretext tasks produce latent spaces where specific varieties follow predictable paths over time, and modeled training trajectories typically closely follow held-out validation data, converging to the end of each sequence even when latent-space gaps introduce noise.
• Grad-CAM points to ripening fruit: Heatmaps from a DINOv2 time-head model emphasize differences in ripeness, highlighting unripe berries as the features distinguishing images taken at different times, suggesting the model uses ripening and growth to estimate time.

Methodology in Plain English

The researchers used a frozen, pre-trained vision transformer to turn each image into a feature vector (the classifier token). On top of that, they attached a small trainable encoder made of two multi-layer perceptrons with ReLU activations, which reduces the feature dimension to n/2 and then n/4. The encoder is trained with several "pretext" tasks at once through separate prediction heads: predict the day of the season (mean squared error loss), predict the cranberry variety (binary cross entropy), predict whether fungicide was applied (BCE), and, for berry crops, predict whether the berry is rotten (BCE). The total loss is the sum of the time, variety and fungicide losses. Training used the Adam optimizer at a learning rate of 0.005 for 8 epochs in PyTorch.

The design intent is that predicting time forces the model to ignore superficial lighting differences (images from the same day should land in the same place) while predicting variety and treatment forces it to keep real biological differences.

After training, UMAP projects the learned latent vectors down to two dimensions for interpretability. To model growth, the researchers compute a velocity for each point as the difference between positions separated by a denoising parameter ε, stack position and velocity vectors, and fit a Bayesian Gaussian mixture model to that joint distribution. At inference, a starting point can be chosen and the distribution conditioned on it, then the velocity is repeatedly integrated to sample a likely future trajectory.

Two scales were analyzed. Patch-based analysis uses 224×224 pixel patches cut from the delineated region after removing the PVC pipe fiducial marker, and suits variety comparison for breeders. Berry-based analysis segments and tracks individual berries, suiting growers focused on yield. For berry tracking, the researchers aligned date-ordered images by extracting XFEAT descriptors between adjacent pairs, generating LighterGlue keypoint correspondences, estimating homographies and warping each image to match its predecessor.

Why This Matters

Impact on research. The work shows a way to get interpretable, low-dimensional growth representations out of large vision foundation models without any manual annotation for fine-tuning, by using crop statistics as self-supervision. It also demonstrates that time prediction alone generalizes to genotypes never seen during training, which is unusual for a model trained on a small domain-specific dataset.

Real-world applications (from the paper's framing):

• Breeders can plot an expected temporal track for a set of cultivars and spot varieties that deviate positively or negatively from desired phenotypes, aiding varietal screening.
• Growers can predict growth over time and monitor responses to treatments such as fungicide, using the predicted-versus-observed track as a decision aid.
• Berry-level tracking supports monitoring berry size, color and ripening rate to help maximize yield.
• Binary rot assessment per berry supports early disease detection and rot-risk management across varieties and treatments.

Industry relevance. Change monitoring in cranberry farming is currently manual and expertise-intensive. The paper's explicit aim is that growers and breeders, who may be skeptical of machine learning, can see exactly how visual evidence maps to an objective crop assessment — a 2D track they can read — rather than an opaque high-dimensional feature. The authors state the approach is general and can be applied to other crops and applications.

Future Directions

• Verify the explainability. The authors state the Grad-CAM results are still in their initial stages and that more work is needed to verify robustness of Grad-CAM on their results as a whole.
• Extend berry-scale tracking to more varieties. Berry analysis covered only varieties C, F and G because other varieties developed dense canopy growth that hindered consistent berry tracking; scaling this up remains open.
• Improve trajectory modeling where latent space has gaps. The paper notes that varieties with larger gaps in latent space are more difficult to model and that this introduces noise in some trajectories, even though trajectories converge at the end.
• Transfer to other crops and applications. The authors present the approach as general but evaluate it only on cranberry; the dataset and code are released to support replication and extension.

Target Audience

This paper is most useful to computer-vision researchers working on self-supervised representation learning, time-series or trajectory modeling, and interpretable latent-space visualization; to agricultural engineers and plant scientists building phenotyping and change-monitoring pipelines; and to agricultural technology practitioners who need a concrete example of adapting vision foundation models to a small, domain-specific crop dataset. Breeders and growers can benefit from the dataset and the interpretability framing, but the method itself requires machine learning background to implement.

Authors’ abstract

Change monitoring is an essential task for cranberry farming as it provides both breeders and growers with the ability to analyze growth, predict yield, and make treatment decisions. However, this task is often done manually, requiring significant time on the part of a cranberry grower or breeder. Deep learning based change monitoring holds promise, despite the caveat of hard-to-interpret high dimensional features and hand-annotations for fine-tuning. To address this gap, we introduce a method for modeling crop growth based on fine-tuning vision transformers (ViTs) using a self-supervised approach that avoids tedious image annotations. We use a two-fold pretext task (time regression and class prediction) to learn a latent space for the time-lapse evolution of plant and fruit appearance. The resulting 2D temporal tracks provide an interpretable time-series model of crop growth that can be used to: 1) predict growth over time and 2) distinguish temporal differences of cranberry varieties. We also provide a novel time-lapse dataset of cranberry fruit featuring eight distinct varieties, observed 52 times over the growing season (span of around four months), annotated with information about fungicide application, yield, and rot. Our approach is general and can be applied to other crops and applications (code and dataset can be found at https://github. com/ronan-39/tlt/).