RF-MatID: Dataset and Benchmark for Radio Frequency Material Identification

AI summary

Overview

Research area: Radio-frequency (RF) sensing and wireless perception for embodied AI, positioned in the paper as a dataset-and-benchmark contribution (arXiv category: Robotics). The work sits at the intersection of RF/radar signal processing, deep learning for signal classification, and robot material perception.

Technical level: Intermediate. The benchmark results and dataset description are accessible, but the sensing setup (vector network analyzer, complex whitening, incidence-angle/stand-off-distance physics) assumes some familiarity with RF measurement.

Scope in one sentence: The paper releases RF-MatID, described as the first open-source, large-scale, wide-band, geometry-diverse RF dataset for fine-grained material identification (16 categories, 5 superclasses, 4–43.5 GHz, 142k samples), together with a multi-protocol benchmark of deep learning models under in-distribution and out-of-distribution conditions.

What This Paper Is About

Material identification — telling whether an object is, say, steel versus aluminum, or granite versus concrete — is normally done with cameras or hyperspectral sensors, which are limited by visual similarity between fine-grained classes and by their inability to sense intrinsic physical properties such as conductivity. RF sensing can expose those intrinsic electromagnetic properties, but the field has lacked a large public dataset and standardized benchmarking. RF-MatID fills both gaps: it provides a public 4–43.5 GHz dataset with controlled geometry variation, plus benchmark results for modern deep learning models tested for robustness under shifts in sensor-object distance and incidence angle.

Key Contributions

1. A first open-source, wide-band, geometry-diverse RF material dataset. RF-MatID covers 16 fine-grained material categories drawn from 5 superclasses (bricks, glass, synthetic materials, woods, stones), spans 4–43.5 GHz (39.5 GHz of bandwidth), and contains 142k samples — 71k in the frequency domain and 71k in the time domain, derived from 71k unique physical measurements. Each sample has 2,048 frequency bins, and the data systematically varies incidence angle (0°–10° at 1° intervals) and stand-off distance (200 mm–2000 mm at 50 mm steps).

2. Investigation of RF data representations and protocol-level applicability. The authors compare raw frequency-domain signals against processed time-domain signals, and evaluate whether raw frequency-domain data can be fed directly to deep models without domain transformation. They also define 5 frequency-allocation protocols, including bands that comply with legal frequency regulations in the United States (P4) and China (P5), and run preliminary experiments on consecutive sub-bands of different bandwidths.

3. A benchmark of learning-based approaches and robustness. The paper establishes a multi-setting benchmark using an MLP, ResNet-50, BiLSTM, vanilla Transformer, TimesNet, Lambert-named RF models Material-ID and AirTac, a hybrid LSTM–ResNet, ConvNeXt, DINOv3, and a custom baseline. Evaluation covers 7 split settings (random, cross-distance, cross-angle) and 3 category divisions (fine-grained, superclass, subclass), reporting accuracy and macro F1 in the main tables and precision/recall in the appendix.

4. Filling a public-availability gap. The comparison table of prior RF material datasets shows every previous dataset listed as not publicly accessible, with benchmark model counts at 0–5 and RF-MatID listed as public with 142,000 samples and 9 benchmark models.

Main Findings

• In-distribution performance is high. The baseline model achieves an average material identification accuracy of 96.83% under the random split (S1) across all protocols and category divisions. In the fine-grained division under Protocol 1 with a random split, reported accuracies reach the high 90s, including 99.84% for the hybrid LSTM–ResNet and 99.57% for the baseline.

• Domain shift is the dominant failure mode. Distance shifts cause an average 24.17% accuracy drop, and angle shifts an average 11.41% drop, relative to the random split.

• Frequency band matters asymmetrically. Millimeter-wave signals (Protocol 2, 30–43.5 GHz) are more robust to distance variation, while centimeter-wave signals (Protocol 3, 4–30 GHz) better tolerate changes in incidence angle.

• Legally constrained bands perform comparably to the full spectrum. Country-specific legal bands (P4, P5) achieve performance comparable to the full 4–43.5 GHz spectrum (P1), indicating RF material identification is feasible under regulatory constraints without significant accuracy degradation.

• Raw frequency-domain data is the more efficient representation. Under Protocol 1 fine-grained classification, the LSTM–ResNet on time-domain data performs comparably to the baseline model on frequency-domain data, but the conversion effort and the much longer 10,240-length time-domain sequence make learning directly from raw frequency-domain data the more optimized and efficient route.

• Dual-channel beats complex-valued input. Representing each frequency bin as separate real and imaginary channels achieves higher classification accuracy than a deep complex network designed for complex-valued inputs, which the authors cite as evidence that phase information can be exploited by learning-based frameworks.

• Model behavior differs sharply by architecture and setting. The MLP performs consistently well but has the largest parameter count (67.20M) and needs redesign of intermediate embedding dimensions across protocols; ResNet-50 and LSTM–ResNet excel under mild shifts (S1, S2-1, S3-1) but degrade sharply under severe shifts (S2-2/3, S3-3); DINOv3 and ConvNeXt are competitive in challenging scenarios such as S3-3, but the low resolution of the RF data hampers stable convergence; sequence-oriented models (vanilla Transformer, BiLSTM, TimesNet) underperform in most settings and are particularly vulnerable to domain shifts; RF-sensing models perform well overall but still drop notably under some out-of-distribution conditions. Averaged across experimental configurations, the baseline reaches about 85% accuracy while the other models perform at approximately 80%.

• **Coarse

Authors’ abstract

Accurate material identification plays a crucial role in embodied AI systems, enabling a wide range of applications. However, current vision-based solutions are limited by the inherent constraints of optical sensors, while radio-frequency (RF) approaches, which can reveal intrinsic material properties, have received growing attention. Despite this progress, RF-based material identification remains hindered by the lack of large-scale public datasets and the limited benchmarking of learning-based approaches. In this work, we present RF-MatID, the first open-source, large-scale, wide-band, and geometry-diverse RF dataset for fine-grained material identification. RF-MatID includes 16 fine-grained categories grouped into 5 superclasses, spanning a broad frequency range from 4 to 43.5 GHz, and comprises 142k samples in both frequency- and time-domain representations. The dataset systematically incorporates controlled geometry perturbations, including variations in incidence angle and stand-off distance. We further establish a multi-setting, multi-protocol benchmark by evaluating state-of-the-art deep learning models, assessing both in-distribution performance and out-of-distribution robustness under cross-angle and cross-distance shifts. The 5 frequency-allocation protocols enable systematic frequency- and region-level analysis, thereby facilitating real-world deployment. RF-MatID aims to enable reproducible research, accelerate algorithmic advancement, foster cross-domain robustness, and support the development of real-world application in RF-based material identification.