Mars-Bench: A Benchmark for Evaluating Foundation Models for Mars Science Tasks

AI summary

Overview

Research area: Computer vision and machine learning for planetary science, specifically benchmarking foundation models on Mars imagery.

Technical level: Intermediate (assumes familiarity with standard vision architectures and evaluation metrics, but explains the Mars science context).

Scope: The paper introduces Mars-Bench, the first standardized benchmark of 20 datasets spanning classification, segmentation, and object detection on Mars orbital and surface imagery, along with baseline evaluations of natural-image, Earth-observation, and vision-language models.

What This Paper Is About

Machine learning research for Mars science has been fragmented: individual studies produce task-specific models and datasets that are undocumented, inconsistently formatted, and impossible to compare against one another. This contrasts sharply with Earth Observation, where benchmarks like Geo-Bench have driven rapid progress by providing consistent evaluation protocols. Mars-Bench addresses this gap by curating and standardizing existing Mars datasets into a unified, machine-learning-ready collection and releasing baseline results across many models and training regimes.

Key Contributions

1. A diverse, unified benchmark: 20 datasets covering classification (binary, multi-class, multi-label), segmentation (binary, multi-class), and object detection, drawn from 2 Mars orbiters (MRO, Mars Odyssey) and 3 rovers (Curiosity, Opportunity, Spirit) across 6 imaging sensors.

2. Scientifically grounded task selection: Datasets target geologic features actively studied by planetary scientists—craters, cones, boulders, landslides, frost, dust devils, atmospheric dust, and landmarks—and were co-developed with domain experts who validated segmentation labels and corrected classification issues.

3. Standardized infrastructure: All datasets ship in ML-ready formats (including COCO, Pascal VOC, and YOLO for detection), with fixed train/val/test splits, few-shot and cross-domain partitioned versions, permissive CC BY 4.0 licensing, loading scripts, and visualization tools.

4. Comprehensive baseline evaluation: Thousands of experiments across architectures, training strategies (from-scratch, frozen feature extraction, full fine-tuning), training-set sizes (1%–100%), Earth-observation foundation models, and proprietary vision-language models, reported with bootstrapped confidence intervals over seven random seeds.

Main Findings

• Transformers lead classification, CNNs still win segmentation. SqueezeNet1.1 underperforms in classification, while ViT-L/16 and SwinV2-B are competitive across datasets. For segmentation, U-Net beats transformer-based SegFormer and DPT on nearly all datasets; DPT is notably unstable with wide confidence intervals.

• YOLO11 dominates detection, but detection is the hardest task. Performance on boulder and dust devil detection is weak overall, attributed to small dataset sizes, few objects per image (often zero or one), grayscale imagery, and low object-background contrast.

• More training data consistently helps, but unevenly. Scaling labeled data from 1% to 100% improves performance in both classification and segmentation, with the rate of improvement varying widely by dataset—evidence of differing task difficulty within the benchmark.

• Earth-observation foundation models transfer poorly. Despite being pre-trained on satellite imagery, SatMAE, CROMA, and Prithvi underperform an ImageNet-pretrained ViT on Mars classification tasks. The authors attribute this to both smaller pre-training corpora and fundamental domain gaps: Martian imagery lacks vegetation, water, and human structures, and has distinct geology, color distributions, and atmospheric conditions.

• Vision-language models are inconsistent. Gemini 2.0 Flash and GPT-4o Mini perform reasonably on classes common in natural images (sand, rock, sky) but collapse on fine-grained geology like crater types and Martian landmarks (F1-scores in the 0.30–0.55 range for those tasks). Smaller VLMs (CLIP, SigLIP, SmolVLM) show the same pattern.

• Mars-specific pre-training is likely necessary. The aggregate evidence—EO models underperforming natural-image models, VLMs failing on domain-specific concepts—supports the conclusion that foundation models trained on Mars data directly would generalize better.

Methodology in Plain English

The authors collected existing Mars datasets from prior publications and mission archives, then reshaped them into consistent formats so that any model can be trained and evaluated on them the same way. They consulted planetary scientists to verify segmentation labels and coordinated with original dataset authors to fix classification problems. For each dataset, they defined fixed train/validation/test splits, preserving original splits where they existed.

To establish baselines, they picked widely used architectures for each task: ResNet101, SqueezeNet1.1, InceptionV3, SwinV2-B, and ViT-L/16 for classification; U-Net, DeepLabV3+, SegFormer, and DPT for segmentation; YOLO11, SSD, RetinaNet, and Faster R-CNN for detection. Each model-dataset combination was tuned via grid search, then retrained with the best hyperparameters on seven random seeds. Results are reported using interquartile mean with bootstrapped confidence intervals and normalized metrics, a methodology borrowed from prior benchmark papers to make scores comparable across heterogeneous tasks.

Beyond standard ImageNet/COCO initialization, they tested Earth-observation foundation models (SatMAE, CROMA, Prithvi) on classification and queried Gemini and GPT with context-rich prompts on six datasets, reformulating segmentation as multi-label classification to fit API-based evaluation.

Why This Matters

Impact on research: Mars science has lacked the evaluation infrastructure that accelerated progress in medical imaging, Earth observation, and NLP. Mars-Bench gives model developers a common yardstick, enables fair comparison between competing approaches, and shifts the field from one-off task-specific models toward reusable foundation models. The dataset partitioning by sensor and mission also creates a controlled setting for studying domain shift and out-of-distribution generalization.

Real-world applications:

• Generating initial global maps of geologic features such as craters and cones, which planetary scientists can refine with minimal manual annotation instead of labeling from scratch.
• Supporting landing-site selection and rover traverse planning by identifying hazards like boulders and steep terrain.
• Monitoring active surface processes such as dust devils, frost, and landslides to understand present-day Martian climate.
• Studying past water and habitability by mapping craters, cones, and terrain types at scale across decades of archived imagery.

Industry relevance: The benchmark's emphasis on label-scarce, highly imbalanced, expert-annotated data and cross-sensor distribution shift makes it a useful stress test for general-purpose vision methods. Researchers working on few-shot learning, domain adaptation, and geospatial foundation models can use it as a real-world evaluation suite outside the usual Earth-centric datasets.

Future Directions

• Mars-specific pre-training. The central open question is whether a foundation model pre-trained on the petabytes of unlabeled Mars imagery would outperform ImageNet- and EO-initialized models by a meaningful margin.

• Expanding and updating datasets. The crater segmentation datasets rely on an older 2010 THEMIS release; newer 2017 imagery could refresh them. Additional geologic features and sensor modalities could broaden coverage.

• Addressing missing georeferencing. Most datasets lack latitude/longitude metadata, preventing spatial distribution analysis and regional generalization studies. Recovering or regenerating this metadata would unlock new research questions.

• Improving detection and small-data performance. Detection tasks and datasets with very few labels (e.g., change detection, boulder segmentation) remain weak points that motivate work on label-efficient and object-detection-specific methods for planetary imagery.

Target Audience

Researchers developing foundation models or vision models for scientific domains, planetary scientists and geologists who want to automate analysis of Mars imagery, remote sensing practitioners interested in domain transfer between Earth and other planets, and machine learning researchers studying distribution shift, label scarcity, and out-of-distribution generalization in real-world settings.

Authors’ abstract

Foundation models have enabled rapid progress across many specialized domains by leveraging large-scale pre-training on unlabeled data, demonstrating strong generalization to a variety of downstream tasks. While such models have gained significant attention in fields like Earth Observation, their application to Mars science remains limited. A key enabler of progress in other domains has been the availability of standardized benchmarks that support systematic evaluation. In contrast, Mars science lacks such benchmarks and standardized evaluation frameworks, which have limited progress toward developing foundation models for Martian tasks. To address this gap, we introduce Mars-Bench, the first benchmark designed to systematically evaluate models across a broad range of Mars-related tasks using both orbital and surface imagery. Mars-Bench comprises 20 datasets spanning classification, segmentation, and object detection, focused on key geologic features such as craters, cones, boulders, and frost. We provide standardized, ready-to-use datasets and baseline evaluations using models pre-trained on natural images, Earth satellite data, and state-of-the-art vision-language models. Results from all analyses suggest that Mars-specific foundation models may offer advantages over general-domain counterparts, motivating further exploration of domain-adapted pre-training. Mars-Bench aims to establish a standardized foundation for developing and comparing machine learning models for Mars science. Our data, models, and code are available at: https://mars-bench.github.io/.