From Images to Decisions: Assistive Computer Vision for Non-Metallic Content Estimation in Scrap Metal

AI summary

Overview

Research area: Applied computer vision and machine learning for industrial quality control, specifically the assessment of non-metallic contamination in scrap metal during railcar unloading at a steel recycling facility.

Technical level: Advanced. The paper assumes familiarity with multi-instance learning (MIL), multi-task learning (MTL), attention pooling, Vision Transformers and Swin Transformers, and CNN backbones such as ResNet, ResNeXt and EfficientNet.

Scope: The paper presents an assistive deep learning pipeline that regresses railcar-level contamination percentage and classifies scrap grade from temporal image layers captured during unloading, and describes its deployment inside a human-in-the-loop acceptance workflow.

What This Paper Is About

Scrap metal quality directly affects energy use, emissions and safety in steelmaking, yet the share of non-metallic inclusions is currently judged visually by human inspectors — an approach the authors describe as subjective and hazardous because of dust, particulate matter, strong magnetic fields and moving machinery. The goal is to build a deep learning pipeline that estimates contamination per percent from images taken during railcar unloading and simultaneously classifies the scrap type, producing a decision aid instead of a purely human judgement.

Key Contributions

1. The authors propose methods based on multi-instance learning (MIL) and multi-task learning (MTL) and report that these achieve quality metrics comparable to those of an inspector.
2. They run experiments on their own dataset in both single-task mode (separate estimation of contamination and scrap class) and multi-task mode, arguing this helps processors optimize quality control, reduce operating costs and minimize waste.
3. They demonstrate that transformer-based models significantly outperform traditional CNN architectures in contamination prediction.
4. They present an end-to-end human-in-the-loop deployment inside the acceptance workflow, including magnet/railcar detection, temporal layer segmentation, a versioned inference service with confidence scores, structured operator overrides, and an active-learning loop.

Main Findings

• Multi-instance learning result: The Swin model used in the MIL setup achieves an MAE of 0.27 and an R² of 0.83 at the railcar level; the table caption specifies a bag of 5 layers for this result.
• Comparison across MIL backbones: EfficientNet 7B reaches MAE 1.43 and R² -0.23; ResNet 50 reaches MAE 0.73 and R² 0.06; ResNeXt 101 reaches MAE 0.56 and R² 0.45; ViT 16 b reaches MAE 0.56 and R² 0.40; ViT 32 b reaches MAE 0.61 and R² 0.31; Swin 2 reaches MAE 0.27 and R² 0.83.
• Inspector comparison: Inspector average is MAE 0.19 ± 0.012 with R² 0.93; the best inspector reaches MAE 0.18 and R² 0.94. The paper states that the metrics of the best model approach the assessment indicators of the inspector.
• Scrap-grade classification (single-task): Swin 2 achieves recall 0.68, accuracy 0.73, precision 0.70 and F1 0.73; ViT achieves recall 0.61, accuracy 0.63, precision 0.64 and F1 0.64; ResNeXt achieves 0.59, 0.61, 0.65, 0.63; EfficientNet achieves 0.60, 0.61, 0.63, 0.61; ResNet achieves 0.53, 0.56, 0.57, 0.55.
• Transformers beat CNNs for grading: Transformer architectures (ViT and Swin) exceed 70% accuracy, surpassing the best CNN by 12 pp, which the authors attribute to the importance of long-range context for distinguishing similar scrap types.
• Multi-task learning result: The Swin-MTL model reaches MAE 0.36, R² 0.78 and F1 0.79 simultaneously, which the authors describe as state-of-the-art on both tasks together.
• MTL model comparison: Swin 2 B reaches MAE 0.36, R² 0.78, recall 0.79, accuracy 0.80, precision 0.80, F1 0.79. ViT B 16 reaches MAE 0.48, R² 0.66, recall 0.65, accuracy 0.68, precision 0.68, F1 0.64. ViT B 32 reaches MAE 0.49, R² 0.65, recall 0.66, accuracy 0.70, precision 0.67, F1 0.63. ResNeXt 101 reaches MAE 0.55, R² 0.58, recall 0.62, accuracy 0.67, precision 0.53, F1 0.56. ResNet 50 reaches MAE 0.74, R² 0.49, recall 0.51, accuracy 0.27, precision 0.41, F1 0.43. EfficientNet 7B reaches MAE 0.80, R² 0.45, recall 0.52, accuracy 0.54, precision 0.51, F1 0.52.
• MTL observation: On average the metrics of all model architectures except Swin are better than those of the MIL approach; ResNet and EfficientNet perform worse on both contamination assessment and scrap metal classification, and transformer-based architectures are better suited to multitask approaches.
• Explainability result: Grad-CAM visualisations show that transformers focus tightly on scrap pieces and ignore dust clouds and background, which the authors say explains their superior regression accuracy relative to CNNs.
• Inspector variability: The spread of individual inspectors relative to consensus reaches up to 2–3 percentage points, while the most stable inspectors show less than 0.5–1.0 percentage points, confirming both an individual offset and different "width" of the scale across inspectors.
• Behavioural effect of the system: The authors report that with the advent of the annotation system and work in the application, inspectors began to pay more attention to the acceptance of scrap metal.

Methodology in Plain English

During unloading, an electromagnet grabs scrap from a railcar in several successive grabs. The ground truth contamination number exists only for the whole railcar, because inspectors give a single holistic assessment rather than rating each grab. The authors therefore treat each railcar as a "bag" containing several layers (magnet grabs), each with its own contamination signal.

A backbone network encodes every layer into a feature vector. An attention module — a two-layer MLP with a Linear–Tanh–Linear structure — produces a score per instance, a softmax turns those scores into weights across the bag, and the weighted sum of features forms the bag representation. A regression head (Linear to 256 units, ReLU, dropout, then Linear to 1) predicts contamination. The backbone architectures are kept intact with their classification layers replaced by identity mappings so that feature embeddings are exposed; all evaluated backbones use a pre-trained ImageNet-1K weighting base.

For the multi-task version, the backbone is shared and branches into two heads: the same regression head plus a classification head (Linear to 256, ReLU, dropout, Linear to the number of classes). The joint objective is the regression loss plus a weighted classification loss, where the weight is selected on the validation set to balance loss scales.

Data preparation uses temporal sequences from fixed high-resolution cameras. Each sequence is segmented into magnet-grab intervals using a magnet position detector, and one or more keyframes are sampled per interval. For annotation, every sample is labelled by three independent annotators under a double-blind scheme, continuous labels are aggregated by the mean, and the standard deviation across annotators is computed as a reliability indicator; samples with standard deviation above 0.4 are flagged for additional verification or adjudication by a senior inspector. Categorical labels are combined by majority vote with expert tiebreak. Splits are made at the railcar level so all layers from one railcar stay in the same partition.

Why This Matters

The work matters because it turns a subjective, safety-hazardous visual judgement into an automated estimate that can be reviewed rather than replaced outright, and it shows that railcar-level labels can be learned from layer-level evidence.

Real-world applications:

• Acceptance and melt-planning decisions at scrap yards and steel plants, where a contamination estimate and grade label feed directly into whether a load is accepted.
• Keeping inspectors out of hazardous zones: in the deployment described, inspectors remain safely inside a dedicated room observing the process remotely instead of standing near the unloading area with dust and moving machinery.
• Stable grading across loads: a shared model reduces the individual offsets and differing "scale widths" the paper documents between human inspectors.
• Continual improvement through the active-learning loop, where operator corrections and uncertain cases are prioritized for expert re-labelling and periodically folded into versioned datasets.

Industry relevance: The paper reports deployment in a live acceptance application with six unloading lines running concurrently, partitioned topics and queues with idempotent writes for exactly-once semantics under retries, and sub-second per-layer inference to keep pace with operations. It also reports a dataset collected from handling over 90,000 tons of metal scrap across over 2000 railcars, and the authors state their pipeline improves the quality of metal scrap delivered to the Electric Arc Furnace and subsequently the quality of rolled steel products.

Future Directions

1. Improve the feature space. The authors argue that better features could let deep learning models approach or surpass human-level performance.
2. Add physical sensing. They suggest incorporating spatial information such as 3D maps to estimate volume, or data from crane scales supporting the electromagnet to obtain both density and volume of scrap metal, since non-metallic inclusions are not magnetic.
3. Address domain generalization. The model was trained and validated exclusively at a single steel recycling facility; variations in lighting, camera setups, scrap composition or unloading protocols at other plants may degrade performance.
4. Publish an industrial dataset. The authors plan to release data on scrap metal contamination and scrap grades from various plants so the community can pursue mutually beneficial human–machine collaboration in metallurgical production. An open question they raise is how to build robust fallback mechanisms when sensor failures or missed frames disrupt MIL bag construction.

Target Audience

This paper is most useful to computer vision and machine learning researchers working on industrial or assistive applications, particularly those interested in multi-instance and multi-task learning with transformer backbones. It is also relevant to metallurgical and recycling process engineers evaluating automated scrap acceptance, and to practitioners designing human-in-the-loop production systems with versioned inference services, operator overrides and active-learning loops.

Authors’ abstract

Scrap quality directly affects energy use, emissions, and safety in steelmaking. Today, the share of non-metallic inclusions (contamination) is judged visually by inspectors - an approach that is subjective and hazardous due to dust and moving machinery. We present an assistive computer vision pipeline that estimates contamination (per percent) from images captured during railcar unloading and also classifies scrap type. The method formulates contamination assessment as a regression task at the railcar level and leverages sequential data through multi-instance learning (MIL) and multi-task learning (MTL). Best results include MAE 0.27 and R2 0.83 by MIL; and an MTL setup reaches MAE 0.36 with F1 0.79 for scrap class. Also we present the system in near real time within the acceptance workflow: magnet/railcar detection segments temporal layers, a versioned inference service produces railcar-level estimates with confidence scores, and results are reviewed by operators with structured overrides; corrections and uncertain cases feed an active-learning loop for continual improvement. The pipeline reduces subjective variability, improves human safety, and enables integration into acceptance and melt-planning workflows.