A graph generation pipeline for critical infrastructures based on heuristics, images and depth data

AI summary

Overview

Research area: Computer vision for critical infrastructure modeling — specifically, graph generation (scene understanding) from photogrammetric data to support digital twins and simulations.

Technical level: Intermediate. Familiarity with object detection, instance segmentation, depth data, and graph concepts helps, but the paper is written in an accessible style and the pipeline's rule-based logic is explained step by step.

Scope: The paper presents and validates a prototypical pipeline that turns RGB images plus depth data from a stereo camera into a relational graph of objects and their physical connections, tested on two synthetic hydraulic systems.

What This Paper Is About

Building virtual copies (digital twins) of physical critical infrastructure such as water or energy plants normally requires expensive 3D laser scanners and specialist expertise. The authors propose a cheaper alternative: use ordinary RGB images and depth data, detect objects with deep learning, and infer how those objects connect using transparent, user-defined heuristics. The goal is a graph in which nodes are objects (pipes, pumps, tanks, valves, sprinklers) and edges are physical connections between them.

Key Contributions

1. A modular graph generation pipeline that separates "pipe" and "non-pipe" objects and processes them with distinct detection, cleanup, matching, and endpoint-estimation strategies before recombining them in a single graph.
2. A rule-based relation inference scheme of seven explicitly defined rules (implemented as functions that take a graph in and return an adjusted graph), chosen deliberately over black-box scene graph neural networks for explainability in high-stakes decision-making.
3. A photogrammetry-based formulation that encodes not just 3D geometry but also relational information between objects, distinguishing it from plain multi-view stereo reconstruction.
4. Validation on two synthetic hydraulic systems of different complexity, reported with graph edit distance as a quantitative metric alongside qualitative comparison.

Main Findings

• System 1 result: For the simpler hydraulic system (one pump, one tank, one valve, one sprinkler, three T-fittings), the predicted graph's overall shape was described as almost identical to ground truth, with a graph edit distance of 4 (cost of 1 for deletion and insertion, cost of 2 for substitution).
• System 2 result: The more complex system (two tanks, two pumps, three valves, four sprinklers) had a graph edit distance of 22 for the entire system, dropping to 10 when errors attributable to the missing sprinklers are disregarded.
• Sprinklers were never detected: In both systems, sprinklers in the images were too small to be detected by the fine-tuned YOLOv8 model. The authors state this is not a shortcoming of the pipeline itself, since the detection model is an interchangeable input.
• Pipe objects sometimes merged incorrectly: In system 1, two subsequent pipe objects were falsely predicted as one — a gold pipe parallel to the y-axis at ground level at (-924, 819) and an olive pipe parallel to the y-axis at ceiling level at (-922, 820), each combining a straight pipe and a 90° bent into a single object.
• Incomplete tank and pipe detection in system 1: The tank (bright green object at (-928, 817)) was approximated in the correct location but not detected entirely, missing a significant part toward the x-axis; a pipe object at ceiling level at (-924, 819) was partly missing.
• System 2 side disconnected: The sprinkler side of the system 2 graph was not connected because a thin vertical pipe at (-6, -8) went undetected. Extra pipe elements between the valves and pumps (for example a small additional pink pipe element at (0, -6)) were caused by YOLOv8 detecting extra elements and subsequent incorrect matching.
• Hyperparameters were tuned per module: Both hydraulic systems used the same settings — np_max_distance = 0.75, p_matching_min_overlap = 0.70, p_threshold = 2.00, p_max_distance = 1.50, and graph_max_distance = 1.50.
• Sensitivity behavior illustrated by figures: The paper presents how results change for np_max_distance values of 0.25, 0.5, 0.75, 1.0, 1.25 and 1.5, and for p_matching_overlap values from 0.1 to 0.9, noting that if np_max_distance is too small, one object may be falsely split into multiple objects.
• Runtime results not reported in the available content: The section is titled "Sensitivity analysis and runtimes," but the provided paper content is truncated before any runtime figures or the discussion in Sections 6 and 7.

Methodology in Plain English

Data first. Because real critical infrastructure sites cannot be photographed for security reasons, the authors built synthetic look-alike scenes in Unreal Engine 5. They used Colosseum (a fork of AirSim) to simulate cameras and produce RGB images plus depth data and the camera position and orientation for each shot. Hydraulic system 1 has one pump, one tank, one valve, one sprinkler and pipes, captured in 16 RGB/depth image pairs at 1920×1080 with a 114° field of view. System 2 adds a second pump and tank, two valves and three sprinklers, captured in 29 image pairs at the same resolution and field of view. Domain randomization was used to vary pipe colors and textures.

Detection models. Both branches of the pipeline use a fine-tuned YOLOv8. For non-pipe objects (pumps, tanks, valves), the largest YOLOv8 pose architecture pre-trained on COCO was fine-tuned for 150 epochs with the Adam optimizer at a learning rate of 10⁻⁴ using 526 images — 280 real-world RGB photos (Canon EOS 7D SLR, 6000×4000) and 246 synthetic Unreal Engine 5 images. It predicts one keypoint for tanks and two keypoints for pumps and valves, marking connection points. For pipes, the largest YOLOv8 instance segmentation architecture was fine-tuned for 200 epochs with Adam at 10⁻⁴ on 1,030 images — 280 real-world and 750 Unreal Engine images — to produce masks instead of keypoints.

Cleaning up detections. Instead of labelling precise object outlines, the authors approximate non-pipe object locations by discarding pixels deeper than the median depth in a bounding box, projecting what remains to a point cloud down-sampled to one voxel per cm³, removing statistical outliers, and clustering with DBSCAN using a 2 cm threshold, keeping the largest cluster. Pipes get the outer two pixels of each mask removed, are projected into 3D with the pinhole camera model, and are cleaned the same way. Pipes that actually belong to non-pipe objects are deleted.

Matching objects across images. Non-pipe detections are merged when the fraction of pairwise voxel distances below np_max_distance (0.75 m) exceeds np_min_percentage, set to 0.1. Keypoints more than two standard deviations from all others are dropped before averaging. Pipe masks are matched by projecting each mask onto other images and computing overlap, then projecting overlaps into 3D, down-sampling to 1 cm³ and clustering with DBSCAN at a minimum 10 cm separation.

Finding endpoints. Straight pipes are distinguished from non-straight ones by comparing the long-to-short side ratio of a rotated bounding box against p_threshold = 2.00. For straight pipes, voxels are binned along the longest side and the means of the first and last bins become the endpoints. For non-straight pipes, the centroid is pulled toward the nearby object with the most voxels within p_max_distance using (1 − p_w) × centroid + p_w × neighbor, with p_w = 0.3, executed twice.

Building the graph. An initial graph is drawn by connecting the closest endpoint pairs first until all remaining distances exceed graph_max_distance = 1.50, with each pair of objects limited to one connection. Seven rules are then enforced by deleting offending edges starting with the largest distance: pumps and valves capped at two connections and tanks at one; pipes capped at three connections; neighbors of the same non-pipe object cannot connect; no single nodes; no cycles; pipes directly attached to pumps classified as Reducer/Expander; and pipes with three neighbors classified as PipeCrossing.

Why This Matters

Research impact. The work argues that heuristic rule-based relation inference is a viable and more transparent alternative to scene graph generation neural networks, which the authors note cannot easily be transferred to new object sets without collecting new data — they point out that many such networks rely on the 3DSSG data set of 1,482 scene graphs with 48k objects and 544k relations, a scale of annotation deemed impractical per use case. The pipeline also frames photogrammetry plus image-based object detection as preferable to point cloud methods for cost, accessibility, and available training data.

Real-world applications:

• Digital twins of water treatment plants, energy plants, and nuclear facilities used for monitoring and optimization.
• Simulation and training for extreme or rare worst-case events in virtual reality, supporting resilience and continuity of services.
• Retrofitting digital models onto older buildings where building information modelling (BIM) was likely never used, requiring geometric capture and object identification from scratch.
• Generalization to other infrastructure sectors named in the paper — energy, healthcare, food, water, transportation, communication, and nuclear systems.

Industry relevance. Laser scanners cost many tens to hundreds of thousands of dollars, are heavy, need expensive processing software, require specialized training, and struggle with reflective surfaces. A stereo-camera pipeline offers a cheaper, more mobile alternative. The deliberate transparency of user-defined rules is framed as a qualification for the high-stakes decision-making that critical infrastructure demands.

Future Directions

• Validating the pipeline on actual real-world scenes, which the authors explicitly state as a remaining requirement since the current validation used synthetic scenes mimicking the real world.
• Improving or replacing the detection model so that small objects such as sprinklers can be detected; the authors already note that the detector is an interchangeable input to the pipeline.
• Developing better endpoint heuristics for ambiguous pipe shapes, identified as a limitation: the current method works for straight and bent pipes but struggles with shapes such as those shown in the ambiguous examples.
• Extending and adapting the rule set to other types of structural critical infrastructure, since the seven rules are stated to be appropriate for hydraulic systems but not sufficient for every infrastructure type, and adding a rule requires only defining and inserting a new function.
• Assessing practical feasibility through the runtime analysis promised in the section title, whose results are not available in the provided content.
• Addressing the errors observed in the results, such as falsely merged pipe objects, extra pipe detections leading to incorrect matching, and undetected thin vertical pipes that leave parts of a system disconnected.

Target Audience

Researchers and practitioners in computer vision, digital twin development, and critical infrastructure protection who need affordable 3D scene understanding without laser scanners. It is also relevant to engineers who need explainable, auditable pipelines for high-stakes settings — for example simulation, resilience planning, and virtual reality training — and to readers interested in photogrammetry-based alternatives to point cloud processing.

Authors’ abstract

Virtual representations of physical critical infrastructures, such as water or energy plants, are used for simulations and digital twins to ensure resilience and continuity of their services. These models usually require 3D point clouds from laser scanners that are expensive to acquire and require specialist knowledge to use. In this article, we present a prototypical graph generation pipeline based on photogrammetry. The pipeline detects relevant objects and predicts their relation using RGB images and depth data generated by a stereo camera. This more cost-effective approach uses deep learning for object detection and instance segmentation of the objects, and employs user-defined heuristics or rules to infer their relations. Results of two hydraulic systems show that this strategy can produce graphs close to the ground truth. While this study focuses on hydraulic systems, the general process can be used to tailor the method to other types of infrastructures and applications. The user-defined rules create transparency qualifying the pipeline to be used in the high stakes decision-making that is required for critical infrastructures.