NegoCollab: A Common Representation Negotiation Approach for Heterogeneous Collaborative Perception

AI summary

Overview

Research area: Computer vision for autonomous driving, specifically heterogeneous collaborative perception, where multiple vehicles share intermediate features to extend perception range.

Technical level: Advanced. The paper assumes familiarity with collaborative perception architectures, intermediate feature fusion, domain adaptation, knowledge distillation losses, and 3D detection metrics such as AP@0.5 and AP@0.7.

Scope: The paper proposes NegoCollab, a framework that derives a shared "common representation" by negotiation among agents with different sensors and frozen perception models, rather than designating one agent's representation as the common one, and evaluates it on OPV2V-H, V2V4Real, and DAIR-V2X.

What This Paper Is About

Collaborative perception lets connected vehicles and roadside units share intermediate features so each sees beyond its own blind spots. When collaborating agents have different, unchangeable sensors and perception models, the features they share live in different domains, and this domain gap degrades fusion. NegoCollab's goal is to close that gap by having a training-time "negotiator" construct a common representation from all participating modalities, so each agent's features only have to be aligned to a middle ground instead of to one specific agent's representation.

Key Contributions

1. It introduces a negotiator that generates the common representation from the local representations of each modality's agent, supervised by a cyclic distribution consistency loss, which reduces the alignment difficulty between local representations and the common representation and offers more diverse, reliable options for the common representation.
2. It introduces a multi-dimensional alignment loss combining distribution alignment loss, structural alignment loss, and pragmatic alignment loss, designed to distill knowledge from the multimodal common representation into each agent's sender.
3. It presents plug-and-play sender–receiver pairs that convert features between each agent's local representation space and the common representation space, eliminating domain gaps during collaboration.
4. Experiments on collaborative perception datasets show NegoCollab achieves state-of-the-art performance among common representation-based methods, outperforming one-to-one adaptation approaches in certain collaborative scenarios.

Main Findings

• Best among common-representation methods on OPV2V-H: On the OPV2V-H dataset (Table 1), NegoCollab reached AP@0.5 of 0.872 (m1m2), 0.911 (m1m3), 0.512 (m2m4), and 0.745 (All), with AP@0.7 of 0.765, 0.854, 0.319, and 0.555 respectively. The paper states NegoCollab achieves the best performance in all test conditions among common representation-based heterogeneous collaboration methods.
• Competitive with one-to-one adaptation: Compared with one-to-one adaptation methods MPDA (AP@0.5 All 0.512, AP@0.7 All 0.435) and PnPDA (AP@0.5 All 0.494, AP@0.7 All 0.424), NegoCollab maintained the best collaborative performance within the initial alliance (m1 and m2) and remained competitive for newly added agents m3 and m4, which did not participate in negotiating the common representation.
• Real-world datasets: On V2V4Real (agents m1 and m3), NegoCollab reached AP@0.5 0.605 and AP@0.7 0.397; on DAIR-V2X (agents m1 and m2), AP@0.5 0.397 and AP@0.7 0.241. The paper reports these results as excellent heterogeneous collaboration performance in real-world environments, though MPDA (0.613/0.400 on V2V4Real, 0.344/0.235 on DAIR-V2X) and PnPDA (0.598/0.385 and 0.443/0.277) remain higher on parts of these benchmarks.
• Homogeneous collaboration gains: In Table 3 on OPV2V-H, NegoCollab achieved AP@0.5 of 0.953 (m1), 0.570 (m2), 0.933 (m3), 0.521 (m4) and AP@0.7 of 0.911, 0.385, 0.888, 0.317. For agents m1, m3, and m4 it even surpassed the "Local" direct feature-sharing baseline (AP@0.5 0.952, 0.930, 0.497 and AP@0.7 0.919, 0.886, 0.322; note that on AP@0.7 for m1 and m4, the reported NegoCollab values 0.911 and 0.317 are slightly below Local's 0.919 and 0.322). The paper attributes improvements to the multi-dimensional alignment loss distilling multi-modal knowledge into local senders.
• Reduced domain gap: Measured with KL divergence (Figure 4), the domain gap between the negotiator-generated common representation and each local representation is reduced by an average of approximately 93.5 compared to designating the protocol agent's representation as the common representation.
• Robustness to localization error: Under Gaussian noise N(0, σ²) applied to both x,y location and yaw angle, the paper reports NegoCollab maintained superior performance on AP@0.5 (Figure 3, described in the text as Figure 4).
• Ablation on training components: On OPV2V-H with agents m1 and m2, the base setting using only distribution alignment loss without the negotiator scored AP@0.5 0.617 and AP@0.7 0.490, while the full combination of negotiator, structural alignment loss, and pragmatic alignment loss scored AP@0.5 0.711 and AP@0.7 0.566.
• Negotiating from different alliances: Table 6 compares common representations negotiated from different initial alliances. For heterogeneous collaboration, the m1m2 alliance produced AP@0.5 0.872/0.770/0.911/0.512/0.745 across m1m2, m3m4, m1m3, m2m4, and All, while the m1m3 alliance produced 0.869/0.832/0.951/0.484/0.830, the m3m4 alliance 0.727/0.840/0.914/0.506/0.737, and the protocol agent 0.792/0.785/0.772/0.499/0.676.

Methodology in Plain English

Each agent keeps its own frozen perception encoder, fusion module, and task head. Around each agent, the method places a sender and a receiver. The sender turns local features into a shared "common representation" for transmission, using a ConvNeXt-based recombiner to enhance features and a size-channel alignment module, followed by an aligner built on fused axial attention to capture global and local dependencies. The receiver does the reverse: a converter built on fused axial attention projects received features back into the agent's own representation space, with its query vector taken from the sender's recombiner output so the local modality guides the transformation, followed by another ConvNeXt recombiner so the result fits the local fusion module.

The distinctive ingredient is the negotiator, used only during training. It feeds standardized local representations into a feature pyramid network, where an estimator at each pyramid level scores how much each modality should contribute, producing an importance matrix. The representations are multiplied element-wise by those weights, averaged across modalities, upsampled, concatenated, and passed through a shrink header to produce the common representation.

Training has two stages. In stage one, the negotiator, senders, and receivers are trained with a cyclic distribution consistency loss (matching the original local feature to the representation reconstructed after a round trip through the common space, plus a standard-deviation term) and a multi-dimensional alignment loss that makes each sender's output match the negotiator's common representation in three ways: distribution (means and standard deviations), structure (relative relations among 9 keypoints per sample, measured with cosine similarity, averaged over 81 entries), and pragmatics (a shared 2D occupancy prediction network trained with focal loss). In stage two, senders are frozen and only receivers are fine-tuned with the downstream collaborative task loss.

The experimental setup uses four collaborating agents plus a protocol agent: m1 and m3 are LiDAR agents, m2 and m4 are camera agents, and the encoder pairs (m1/m3, m2/m4) differ. The protocol agent, m1, and m3 use LiDAR while m2 and m4 use cameras. Encoders are PointPillars (protocol, m1), Lift-Splat with EfficientNet at 384-pixel image height (m2), SECOND (m3), and Lift-Splat with ResNet50 at 336-pixel image height (m4). Training starts with a homogeneous model for each agent type, then negotiates the common representation from the m1–m2 alliance, then adds m3 and m4. Experiments used a single RTX 4090 GPU with initial learning rate 0.001 and the Adam optimizer; the first training phase took roughly 4–12 GPU hours at about 23GB memory, and the second about 2–5 GPU hours at roughly 14GB memory, depending on agent architecture. Datasets: OPV2V-H (73 scenes, 6 road types, 9 cities, 36K LiDAR point cloud frames, 12K RGB frames, 12K depth frames, 230K annotated 3D bounding boxes), DAIR-V2X (9K frames, one vehicle and one roadside unit, 300-channel RSU LiDAR and 40-channel vehicle LiDAR, 1920x1080 cameras), and V2V4Real (20,000 LiDAR scans, 240,000 annotated 3D bounding boxes, five vehicle classes).

Why This Matters

Research impact: The paper reframes the common representation in heterogeneous collaboration as something to be negotiated rather than something borrowed from one agent, and shows that alignment difficulty depends on which agent's representation is chosen. It also demonstrates that a single negotiated representation can be reused by newly joining agents with different sensors and encoders, which matters for incremental fleet deployments. The multi-dimensional alignment loss (distribution, structure, pragmatics) offers a template for distilling multimodal knowledge into modality-specific encoders without retraining them.

Real-world applications:

• Vehicle-to-vehicle and vehicle-to-infrastructure cooperative 3D detection, where fleets mix camera-only and LiDAR-only vehicles with different sensor channel counts.
• Intersection and blind-spot warning systems, where roadside units and vehicles with different vendors' perception stacks must share features.
• Mixed-fleet retrofitting, where newer vehicles with different sensors join a group that already negotiated a shared representation.
• Robustness-oriented deployment, since the method is tested under Gaussian pose noise on x, y, and yaw and on real-world datasets V2V4Real and DAIR-V2X.

Industry relevance: The method treats perception encoders, fusion modules, and task heads as frozen, which addresses the "immutable heterogeneity" constraint that makes retraining deployed safety-critical models impractical. It requires only plug-and-play sender–receiver pairs and reports low training cost compared with one-to-one domain adaptation, and reaches competitive or better accuracy within an initial alliance.

Future Directions

• The paper explicitly names one limitation: once the common representation is negotiated, it becomes fixed, and aligning new agents to it inevitably causes greater information loss. Making the common representation generalize better to new agents is stated as future work.
• The paper raises the question of how to choose which agents should negotiate the common representation, noting that heterogeneous agents form multiple collaborative groups and that the negotiation mechanism allows free selection; the truncated appendix reportedly contains detailed results on negotiating from different initial alliances, including m1m3, m1m2, and m3m4.
• Extending the evaluation across more modalities and larger agent populations than the four configured agent types plus protocol agent.
• Reducing the residual gap on real-world benchmarks, where one-to-one adaptation methods such as PnPDA still lead on DAIR-V2X (AP@0.5 0.443 and AP@0.7 0.277 versus NegoCollab's 0.397 and 0.241).

Target Audience

Researchers and engineers working on collaborative perception, multi-agent autonomous driving, and heterogeneous sensor fusion; practitioners dealing with mixed fleets where perception models cannot be retrained; and readers interested in cross-modal representation alignment and knowledge distillation losses applied to 3D detection. Students with a background in 3D object detection and feature fusion will find the framing accessible, though the notation and loss formulations require intermediate-to-advanced familiarity with the field.

Note: the training-setup ablation in Table 4 and the domain-gap figure numbering in the provided text are partially garbled in the source; figures cited as "Figure 3" for robustness and "Figure 4" for domain gaps are referred to interchangeably in the supplied text, and the Table 4 checkmark alignment is ambiguous.

Authors’ abstract

Collaborative perception improves task performance by expanding the perception range through information sharing among agents. . Immutable heterogeneity poses a significant challenge in collaborative perception, as participating agents may employ different and fixed perception models. This leads to domain gaps in the intermediate features shared among agents, consequently degrading collaborative performance. Aligning the features of all agents to a common representation can eliminate domain gaps with low training cost. However, in existing methods, the common representation is designated as the representation of a specific agent, making it difficult for agents with significant domain discrepancies from this specific agent to achieve proper alignment. This paper proposes NegoCollab, a heterogeneous collaboration method based on the negotiated common representation. It introduces a negotiator during training to derive the common representation from the local representations of each modality's agent, effectively reducing the inherent domain gap with the various local representations. In NegoCollab, the mutual transformation of features between the local representation space and the common representation space is achieved by a pair of sender and receiver. To better align local representations to the common representation containing multimodal information, we introduce structural alignment loss and pragmatic alignment loss in addition to the distribution alignment loss to supervise the training. This enables the knowledge in the common representation to be fully distilled into the sender.