BackDFL: A Unified Benchmark For Backdoor Attacks and Defenses In Decentralized Federated Learning

AI summary

Overview

Research area: Security and privacy in decentralized federated learning (DFL) — specifically backdoor attacks and Byzantine-robust defenses evaluated over peer-to-peer communication graphs.

Technical level: Advanced. The paper assumes familiarity with federated learning aggregation, poisoning threat models, graph topologies, and backdoor evaluation metrics (ASR, accuracy, durability).

Scope: The paper introduces BackDFL, a reproducible benchmark suite and empirical study comparing thirteen FL/DFL backdoor defenses against six targeted backdoor attacks across seven datasets and multiple communication graph topologies.

What This Paper Is About

Decentralized Federated Learning removes the central parameter server and lets clients exchange model updates directly with their peers, which is meant to eliminate single points of failure and trust dependencies. The paper argues that this architectural shift has been under-studied from a security perspective: existing robustness claims rest on simplified threat models, non-adaptive attackers, inconsistent topologies, and ad hoc training configurations, so the true vulnerability of DFL to backdoors is unknown. BackDFL is the authors' answer — a unified, configuration-driven benchmark for measuring backdoor attacks and defenses in fully decentralized settings under realistic and adaptive adversaries.

Key Contributions

1. A unified, reproducible benchmark for backdoor robustness in fully decentralized FL. BackDFL is implemented in Python/PyTorch with a modular architecture (configuration, data and model, experiment flow, attack and defense, base FL, and output layers) driven by YAML configuration files, and is released publicly at https://github.com/mohaminemed/BackDFL.

2. An extensive comparative evaluation of thirteen defenses showing their failure modes. The evaluated set combines five DFL-specific methods (UBAR, SCCLIP, BALANCE, DFL-Dual, ABALANCE) and eight centralized-FL defenses adapted to local, client-side aggregation (DeepSight, FLAME, SPP, MMAD, Norm-Clip, Weak-DP, M-Krum, Trim). Framework text also states the broader implementation integrates fourteen defenses, described as nine adapted FL backdoor defenses and five DFL-specific robust methods.

3. Harmonized attack definitions and an adaptive threat model. The benchmark integrates six well-established targeted backdoor attacks (fixed trigger, optimized trigger, and gradient manipulation families — including DBA, Scaling, Neurotoxin, A3FL, and IBA) plus five classical untargeted attacks (Label Flipping, Feature, Gauss, Krum, and Trim).

4. ABALANCE, an adaptive extension of the BALANCE defense. Instead of BALANCE's fixed acceptance rule, ABALANCE uses a threshold that reacts to both the distribution of received neighbor distances and their evolution over rounds: τ_adaptive(t) = min(median(d_ij(t)) + σ, τ_prev) · 1/(1 + γσλ(t)), where σ is the median absolute deviation (MAD) of the received distances and τ_prev is the previous round's threshold.

Main Findings

• Decentralization fundamentally changes threat dynamics. Under identical settings (Neurotoxin on HAR, 15% malicious clients), centralized FL methods all keep accuracy at or above 0.94, while in DFL even strong defenses degrade; Weak-DP, DeepSight, and SPP collapse to an ASR of approximately 1.0 with maximal durability. The paper attributes this to each client seeing only a limited neighborhood and to local reference models themselves becoming progressively contaminated, making adversarial updates look consistent with the neighborhood.

• Defenses effective in centralized FL do not transfer. In the centralized comparison, M-Krum, FLAME, and SPP suppress the attack with ASR of roughly 0.16 to 0.21 and zero durability, whereas Norm-Clip, Weak-DP, and DeepSight show ASR up to 0.84 with maximum durability despite preserved accuracy.

• Attack strength ranks IBA > A3FL > Neurotoxin. Mean ASRs are 0.57 / 0.33 / 0.22 under IID and 0.74 / 0.52 / 0.35 under non-IID partitions. IBA causes an ASR ≥ 0.500 failure in 75% of defense–dataset–partition combinations, versus 35% for A3FL and 21% for Neurotoxin. The paper reports static attacks (BadNet, DBA, Scaling) as generally less effective in this setting and omits them for brevity.

• IBA breaks every evaluated defense on CIFAR-10. All defenses exceed 0.677 ASR on CIFAR-10 in both IID and non-IID partitions, including ABALANCE and FLAME. The gap is attributed to a dataset-level vulnerability: CIFAR-10's lower per-class margin under a 10-way task, which IBA's optimized trigger exploits independently of the aggregation rule.

• On GTSRB, protection narrows under heterogeneity. Under IID GTSRB, M-Krum, ABALANCE, FLAME, MMAD, UBAR, and DFL-Dual keep IBA below 0.10 ASR. Under non-IID GTSRB, only ABALANCE and UBAR remain below the 0.500 failure threshold.

• Consistency versus IID dominance. Averaged over all attacks, datasets, and partitions, ABALANCE and FLAME are the most consistent defenses (mean ASR 0.347 and 0.341), but under IID data alone DFL-Dual and M-Krum are marginally lower. That IID advantage disappears under non-IID data, where M-Krum and MMAD post the largest degradations of any defense, followed by DFL-Dual — consistent with their reliance on inter-client similarity or tightly clustered updates.

• SCCLIP is the weakest defense overall, with a mean ASR of 0.685, indicating that self norm clipping alone is insufficient against adaptive triggers. DeepSight, BALANCE, and SPP form a middle tier.

• Aggressive filtering does not reliably buy robustness. DeepSight has the worst Min.ACC on CIFAR-10 IID across all three attacks (0.810) while its mean ASR (0.450) is only mid-tier; UBAR posts the worst Min.ACC under non-IID.

• Dataset and task properties govern backdoor persistence. With A3FL on MNIST, nearly every defense drives durability to 0 and final ASR below 0.22, with only Weak-DP failing. Under the same attack on FEMNIST, M-Krum, ABALANCE, BALANCE, and MMAD block the backdoor from round one (durability = 0), clipping- and similarity-based defenses leave ASR pinned near 1.00 for the full 49-round window, and FLAME, UBAR, and DFL-Dual suppress the attack only after a long delay. On CIFAR-100 with IBA, most defenses lower Max.ASR relative to CIFAR-10 yet durability remains above 40 for nearly all of them.

• Temporal alignment cuts both ways. On HAR under Neurotoxin, the alignment of benign gradients enables low-norm, temporally consistent poisoning that bypasses SCCLIP (final ASR = 0.74), Weak-DP (0.65), and DeepSight (0.33), all retaining full durability, whereas Trim, BALANCE, MMAD, and DFL-Dual suppress the same attack to final ASR ≤ 0.20. UBAR's high false-positive rate degrades Min.ACC to 0.76 while still failing to fully contain the attack.

• Graph topology governs both attack amplification and recovery. Without any defense on HAR under Neurotoxin (attack window [25,50]), peak ASR ranges from 65.3% on regular-(50-10) to 97.8% on ring, with barabasi, erdos, and watts clustering around 88–95%. Only ABALANCE and UBAR suppress the backdoor on all six topologies (0–6 rounds, final ASR ≤ 21%). Trim suppresses instantly (0 rounds) on barabasi, erdos, regular-(20-7), regular-(50-10), and ring, but never recovers within the training horizon on watts (36.3 rounds, censored, final ASR 43.9%). Norm-Clip, DeepSight, Weak-DP, SPP, BALANCE, and DFL-Dual remain unsuppressed (50+ rounds, censored) on shortcut-rich topologies but recover on the sparse ring (Norm-Clip: 24 rounds, DeepSight: 22 rounds, SPP: 16 rounds).

Methodology in Plain English

The authors build a modular software framework rather than running a single experiment. Datasets, models, attacks, defenses, and experiment flows are separate modules, and each run is defined by a YAML configuration file, so the same attack or defense can be swapped in without rewriting core code.

They model DFL as an undirected communication graph G = (V, E). Each client trains locally on its own data, then blends its own updated model with an aggregate of its neighbors' models using a mixing ratio α. The default graph is a regular graph with N = 20 clients and degree k, denoted regular-(20-10), with all clients participating in every round, and they also test regular-(20,7), regular-(50,10), ring-(20,7), watts-(20,7), barabasi-(20,7), and erdos-(20,7) graphs.

The threat model is an adaptive adversary controlling a fraction β_M of clients, default 15%. Malicious clients can inject poisoned samples into their local data or manipulate the model updates they share. The attacker knows the compromised clients' training data plus the aggregation rules and hyperparameters, but has no prior knowledge of the communication graph or neighborhood structure. The defender does not know which clients are malicious, has no visibility into the attack strategy, and does not know the global or local malicious ratios. The topology is fixed and cannot be altered by the attacker.

Data heterogeneity is introduced with a Dirichlet partition using concentration parameter α = 0.5. Clients train with mini-batches of 32; evaluation uses batch size 256; the default learning rate is 0.01 and the mixing ratio α is 0.5, with different initial models per client. Poisoning covers 25% of local data for all attacks, and poisoned samples are relabeled to the target class. Two local training regimes are compared: one epoch per round (communication-intensive) and five local epochs per round (communication-efficient). Experiments run on two identical NVIDIA L40S GPUs.

Evaluation reports Main-task accuracy as Avg.ACC and Min.ACC over benign clients, attack success rate as Max.ASR across benign clients, and durability as Final.ASR plus the backdoor Lifespan — the number of communication rounds required for Max.ASR to drop below a predefined threshold (e.g., 0.5). The Attack Section for RQ2 uses a [30,60] attack window, RQ4 uses [25,50], and three seeds are used for topology experiments: 21, 42, and 84. Sensitivity analyses on malicious rate, data heterogeneity, and mixing ratio are said to be provided in Appendix 0.B, which is not included in the available content.

Why This Matters

Impact on research. The paper challenges the assumption that Byzantine-robust DFL is defensible, showing that a 15% malicious participation rate is sufficient to break state-of-the-art methods, and that failure rates, durability, and recovery speed depend heavily on dataset, partition type, and graph topology. It also documents a reproducibility problem in the field: most DFL defenses (BALANCE, UBAR, DFL-Dual) lack public, reproducible implementations, with only a few exceptions such as LEARN and SCCLIP, and prior studies vary widely in topology, local training schedules, optimizers, and hyperparameters — making cross-paper robustness claims hard to compare.

Real-world applications mentioned or implied by the paper's framing:

• Connected and autonomous vehicles collaboratively learning perception models from locally captured images or video to detect traffic signs, lanes, and pedestrians.
• Internet of Things ecosystems, described as large-scale, dynamic environments where DFL's fault tolerance is attractive.
• Smartphone-sensor human activity recognition, represented by the HAR dataset of 561-dimensional feature vectors in the evaluation.
• Traffic sign recognition specifically, represented by the GTSRB dataset with a 43-class output layer.

Industry relevance. Any deployment that replaces a trusted aggregator with peer-to-peer model exchange — vehicular fleets, IoT fleets, cross-organization collaborative learning — inherits the vulnerabilities documented here. The finding that a lightweight adaptive rule (ABALANCE) can outperform more computationally expensive centralized defenses while suppressing backdoors across all six tested topologies gives practitioners a low-computation option, and the topology results give system designers a concrete reason to treat graph structure as a security parameter rather than an incidental detail.

Future Directions

• Defenses that generalize across topologies. Only ABALANCE and UBAR suppressed the backdoor on every topology tested, and the paper concludes there is a need for adaptive mechanisms that maintain robustness under diverse networks.

• Closing the difficulty gap. The topology results are obtained on HAR under Neurotoxin, described as one of the lightest tasks paired with one of the least stealthy attacks; under stealthier attacks (IBA) and on harder tasks (GTSRB and CIFAR-10), all defenses fail regardless of topology. Whether any defense can hold under that combination is left open.

• Attackers that know the graph. The threat model deliberately withholds graph knowledge from the attacker. Whether a graph-aware adaptive adversary performs even better, and whether any defense survives it, is not addressed.

• Addressing the reproducibility gap. The paper notes that most DFL defenses lack publicly available, reproducible implementations; making these available and standardizing topologies, local training schedules, and hyperparameters is a prerequisite for comparable robustness claims.

• Scale and sensitivity beyond the defaults. The default configuration is 20 clients on a regular graph. Sensitivity analyses on malicious rate, data heterogeneity, and mixing ratio are deferred to Appendix 0.B, which is outside the available text, so the boundary conditions of these results are not reported here.

Target Audience

Researchers and graduate students working on federated learning security, poisoning and backdoor attacks, and Byzantine-robust aggregation; practitioners designing peer-to-peer or serverless collaborative learning systems for vehicular networks and IoT; and benchmark or reproducibility-focused engineers who need a configurable testbed for comparing attacks and defenses across datasets, models, and communication topologies. Readers should already be comfortable with federated learning aggregation, attack success rate and accuracy metrics, and graph-based network models.

Authors’ abstract

Decentralized Federated Learning (DFL) promises trust-free collaborative learning by replacing the centralized parameter server with peer-to-peer model exchange. However, this architectural shift fundamentally reshapes the threat landscape. Without globally coordinated aggregation, DFL becomes particularly susceptible to backdoor attacks, in which malicious participants implant persistent hidden behaviors while maintaining high clean-task performance. In this paper, we argue that the robustness of DFL has been significantly overestimated. Existing studies rely on simplified threat models, non-adaptive adversaries, fragmented evaluation protocols, inconsistent communication topologies, and ad hoc training configurations, leading to an incomplete understanding of DFL security. To address these limitations, we present BackDFL, a unified benchmark for systematically evaluating DFL under realistic and adaptive backdoor attacks. Through extensive experiments, BackDFL exposes critical failure modes of decentralized learning. Our results demonstrate that both state-of-the-art Byzantine-robust DFL methods and adapted FL backdoor defenses fail under modest malicious participation rates (as low as 15%), especially in heterogeneous settings, while their robustness varies substantially across communication graph topologies.