Walk the Lines 2: Contour Tracking for Detailed Segmentation

AI summary

Overview

Research area: Computer vision, specifically detailed object segmentation via contour tracking, applied to infrared (IR) ship imagery and a range of RGB object categories.

Technical level: Advanced. The paper assumes familiarity with convolutional neural networks, object contour detection, non-maximum suppression, active contour methods, and segmentation metrics such as IoU, precision, and recall.

Scope: The paper presents WtL2, an extension of the Walk the Lines contour tracking algorithm that adapts to IR ship segmentation and generalizes to varied RGB objects, achieving state-of-the-art results among contour-based methods on the subset of images where a closed object contour is successfully formed.

What This Paper Is About

Detailed segmentation of objects such as ships, including fine structures like antennas and masts, is difficult for standard mask-based deep learning methods, which tend to produce blob-like masks. The original Walk the Lines (WtL) algorithm solved this for RGB ships by tracking along an object contour instead of thinning it, but it only worked on that single category and that single imaging domain. WtL2's goal is to broaden this approach to infrared ship imagery and to a wide variety of RGB objects, while producing high-quality, closed, 1-pixel-wide contours that can be binarized into segmentable areas.

Key Contributions

1. Applicability of WtL2 to IR ship segmentation. The authors demonstrate detailed segmentation of IR ship images by adapting only the input object contour detector (RefineContourNet, RCN) to the IR domain, producing RCN-IR, while leaving the core contour tracking CNN unchanged.

2. Generalization to various RGB objects. WtL2 extends beyond ships to categories including cars, dogs, deer, giraffes, and elephants, achieved by retraining the tracking CNN and modifying the binarization step, which was originally tailored to ships.

3. A new binarization method (Algorithm 1). Instead of relying on the ship-specific assumption of a visible waterline, WtL2 uses a general thresholding procedure that starts from the WtL contour and searches for a separation line that connects object and background.

4. New datasets and baselines for comparison. The authors release the DOC (Detailed Object Contour) dataset — 90 COCO test-dev images with manually refined annotations plus 50 validation images with ground truth — the ESIRRID dataset, 10 Detailed IR Ship Contour (DIRSC) ground truth images, and an RN-IR baseline (RefineNet with a ResNet101 backbone, fine-tuned on 227 self-training labels).

Main Findings

• WtL2 nearly doubles WtL's IoU on the DOC validation set. On 50 DOC validation images, WtL closed shapes in 28 cases (56%) with an IoU of 44.57, while WtL2 closed shapes in 40 cases (80%) with an IoU of 75.74. NMS closed only 2 shapes (4%), with no IoU reported.

• WtL2 beats the state of the art among contour-based methods when a closed contour forms. On 50 refined COCO test-dev images, E2EC achieved 89.31 precision, 89.71 recall, and 84.50 IoU overall, compared to WtL2-seg at 89.55 / 81.02 / 75.74. On the 40 images where WtL2 formed a closed object contour, WtL2-seg reached 93.58 precision, 94.83 recall, and 89.01 IoU, surpassing E2EC's 89.40 / 89.84 / 84.06 on the same row.

• IR ship results show extreme variance but high peaks. Across 10 DIRSC validation images, WtL2-seg-IR averaged 69.35 precision, 58.11 recall, and 52.01 IoU overall, versus the RN-IR baseline at 81.44 / 92.54 / 76.83. On the six-image subset where WtL2 formed closed contours (images 1, 2, 3, 4, 5, 9, 100% closed), WtL2-seg-IR reached 89.10 / 96.84 / 86.67, slightly outperforming RN-IR's 89.73 / 95.99 / 86.60 on that row.

• Only 60% of IR images produced a closed shape. Binary WtL2 formed closed contours on 60% of the 10 DIRSC images, and NMS closed none (0%). Complete IoU failures occurred on some images, which the authors attribute to the method's robustness limitations rather than to the IR adaptation itself.

• Recall is consistently high. The authors highlight high recall as a property of the WtLs, visible in both the DOC and IR results.

• The core tracking algorithm transferred to IR without modification. Only the object contour detector was retrained for IR; the tracking CNN was unchanged, which the authors present as evidence of the core algorithm's robustness.

Methodology in Plain English

The pipeline starts with an object contour detector (RefineContourNet), which produces a "soft" contour map with thick, probabilistic edges. Standard practice would apply non-maximum suppression (NMS) to thin the contour, but NMS introduces gaps. Instead, WtL2 deploys many simultaneous "trackers" — hundreds, or thousands depending on contour complexity — that walk pixel by pixel around the object. Each tracker crops a patch centered on its current point, rotates it based on the previous direction of travel, and feeds a 7×7×4 patch (image plus soft contour) into a small tracking CNN. The CNN outputs a direction in degrees (for example, -12.054°), which is rounded to one of eight discrete directions: -135°, -90°, -45°, 0°, 45°, 90°, 135°, or, rarely, ±180°. The pixel step size is stochastically chosen with a preference for 1 over 2 or 3. This produces a connected, 1-pixel-wide closed contour, which is then binarized into a filled area usable as a segmentation mask. Tracking typically takes a few tens of seconds per image.

For IR, the authors adapted the input side: they used a highly unbalanced RGB ship dataset to emphasize ships, adjusted the IR intensity channel to resemble typical color channel values, generated labels with a Conditional Random Field method, ran self-training up to 227 images, and trained RCN-IR. For general RGB objects, they retrained the tracking CNN and replaced the ship-specific binarization (which assumed a waterline) with Algorithm 1, which picks a separation line orthogonal to the contour whose endpoints touch the two largest enclosed areas (object and background), then thresholds to close the contour. The baseline RN-IR uses RefineNet with a ResNet101 backbone, fine-tuned on the same 227 self-training labels, allowing direct comparison between conventional deep learning and WtL2.

Why This Matters

Impact on research: The paper argues that detailed, high-quality contours remain a gap that mainstream mask-based segmentation and even foundation models like SAM do not fully address in niche domains. WtL2 targets applications requiring extreme detail, and the authors position it as a complement to existing techniques rather than a replacement. Releasing DIRSC ground truth, ESIRRID, and the DOC subset addresses a documented shortage of comparable public data for IR ship segmentation.

Real-world applications:

• Maritime navigation and traffic safety, where IR imagery improves visibility and detailed ship shapes (antennas, superstructures, bridge wings) matter.
• IR ship segmentation in coastal and harbor monitoring, where few methods and datasets exist.
• High-quality sample generation for training data in rare or unusual object categories such as deer, giraffes, and elephants.
• Specialized fields needing fine detail in foreground-background scenarios, as the method extracts features like a motorcyclist's helmet shield and tire treads, an elephant's belly and tail, and a giraffe's hump, ears, mouth, neck, tail, and legs.

Industry relevance: The method is presented as a specialized tool for applications where contour quality is the bottleneck, particularly in the underexplored IR maritime domain where model weights are often not publicly available. Its speed — a few tens of seconds per image with all trackers running simultaneously — makes it practical for offline or high-quality-sample production rather than real-time deployment.

Future Directions

• Improving robustness. The authors report "almost total failure" on some images and complete IoU failures in the IR set, so making closed-contour formation more reliable is a clear open problem.
• Avoiding the need for closed contours. The method only shines once a closed object contour exists, so handling the cases that currently fail would expand its applicability substantially.
• Extending beyond the current domains. The paper mentions "specialized fields not yet fully explored," leaving open which other imaging modalities or object categories could benefit.
• Benchmarking against additional methods. The authors note that few methods are similar to theirs, limiting comparison to E2EC for color and their self-built RN-IR baseline for IR.

Target Audience

Researchers and practitioners in computer vision working on segmentation, particularly those focused on IR imagery, maritime perception, or niche and rare object categories where detailed boundaries matter more than overall average IoU. It is also relevant to engineers building data-generation pipelines for high-quality training samples, and to readers interested in alternatives to NMS thinning and active-contour refinement such as DeepSnake and E2EC. Because the paper assumes knowledge of CNN-based detection, CRF labeling, self-training, and IoU-based evaluation, beginners would need substantial background reading.

Authors’ abstract

This paper presents Walk the Lines 2 (WtL2), a unique contour tracking algorithm specifically adapted for detailed segmentation of infrared (IR) ships and various objects in RGB.1 This extends the original Walk the Lines (WtL) [12], which focused solely on detailed ship segmentation in color. These innovative WtLs can replace the standard non-maximum suppression (NMS) by using contour tracking to refine the object contour until a 1-pixel-wide closed shape can be binarized, forming a segmentable area in foreground-background scenarios. WtL2 broadens the application range of WtL beyond its original scope, adapting to IR and expanding to diverse objects within the RGB context. To achieve IR segmentation, we adapt its input, the object contour detector, to IR ships. In addition, the algorithm is enhanced to process a wide range of RGB objects, outperforming the latest generation of contour-based methods when achieving a closed object contour, offering high peak Intersection over Union (IoU) with impressive details. This positions WtL2 as a compelling method for specialized applications that require detailed segmentation or high-quality samples, potentially accelerating progress in several niche areas of image segmentation.