Federated Learning and Secure Aggregation

A coordinator, a set of institutions that never share raw records, an aggregation protocol, the threats that protocol invites, and the agreement that says who answers for any of it.

The scale at which that question arises is not hypothetical. The largest federated study yet run built an automatic tumour-boundary detector for glioblastoma, across hospitals on six continents. Nature Communications published it in 2022. From the abstract: “Here we present findings from the largest FL study to-date, involving data from 71 healthcare institutions across 6 continents, to generate an automatic tumor boundary detector for the rare disease of glioblastoma, utilizing the largest dataset of such patients ever used in the literature (25,256 MRI scans from 6,314 patients).” Against a publicly trained model, that federation improved delineation of the surgically targetable tumour by 33%, and of the whole tumour extent by 23%.

Seventy-one institutions on six continents. 25,256 scans from 6,314 patients. One coordinator deciding the rounds and the client sample for all of them. Every box below is a different organisation with its own lawyers.

Federated learning coordinates model training across distributed data holders, often by aggregating locally computed updates. It reduces raw-data centralization. It does not provide privacy, fairness, security, or governance automatically. Secure aggregation can prevent the coordinator from seeing individual updates while revealing an aggregate under stated assumptions. Additional risks include update leakage, malicious clients, model poisoning, non-IID data. They include participation bias, dropout, unequal benefit, and difficult deletion or attribution.

Both halves of the machinery are old enough to be standard. Federated averaging was published in 2017, by McMahan and co-authors. A secure aggregation protocol arrived the same year, from Bonawitz and co-authors, and it holds up even when an arbitrary subset of clients drops out mid-round. Neither result removes the coordination problem sitting above the cryptography. The server still chooses the objective, the rounds and the client sample.

The phrase doing the work there is “under stated assumptions”. The assumption is an honest-but-curious server. In 2022 a paper dropped it. Pasquini and colleagues write: “In this work, we show that a malicious server can easily elude secure aggregation as if the latter were not in place.” The method sends inconsistent model versions to different clients, so a target client's update can be isolated inside the aggregate. It infers information about individual training sets independently of how many users take part. The count of participants is the thing an aggregate is supposed to hide a client behind. Here it stops protecting anyone. Their canary-gradient attack is about 99% accurate for batch sizes up to 128.

So a protocol that hides an individual update from a curious coordinator does not bind a dishonest one. And the party that would have to be dishonest is the party that chooses the objective, the rounds and the client sample.

Twenty institutions trained the EXAM model to predict clinical outcomes in patients with COVID-19 without exchanging raw records. Nature Medicine published the result in 2021. The abstract states: “EXAM achieved an average area under the curve (AUC) >0.92 for predicting outcomes at 24 and 72 h from the time of initial presentation to the emergency room, and it provided 16% improvement in average AUC measured across all participating sites and an average increase in generalizability of 38% when compared with models trained at a single site using that site's data.”

Read the arithmetic rather than the headline. The 16% is an improvement in average AUC measured across all participating sites. The 38% is an average increase in generalizability. Both are averages taken over 20 institutions. An average of that shape cannot tell a board whether its own hospital gained, held level, or lost. A site whose patients were served worse by the federated model sits inside the same 16%, invisible. The number was computed by averaging across the sites, not reported for each of them. A strong aggregate does not make the per-site picture knowable. It has to be published site by site, or it is not evidence a participant can act on.

Centralized training, federated learning, and secure aggregation move the risk rather than removing it. Each leaves a different party able to see something.

The gap between the first two columns has been measured. A brain-tumour segmentation model was federated across 10 institutions, then set against the same data pooled in one place. Sheller and colleagues reported the result in Scientific Reports in 2020: “We show that federated learning among 10 institutions results in models reaching 99% of the model quality achieved with centralized data, and evaluate generalizability on data from institutions outside the federation.”

Hold that 99% against the first column. In that task, at that scale, the accuracy a federation gives up by not pooling was one percentage point of model quality. Which is the point. If the choice between the columns were a utility choice, it would already be settled. It is a visibility choice. Each column decides who gets to see what — the operator of the central environment, the coordinator, or nobody in particular.

Collaboration purpose comes first and exit terms come last. Everything between them is a threat model.

The vocabulary for the middle does not have to be invented at the table. A standard already covers federated machine learning: IEEE 3652.1-2020, published in 2021, covering architecture, categories, performance evaluation and associated regulatory requirements. Its scope reads: “A blueprint for data usage and model building across organizations and devices while meeting applicable privacy, security and regulatory requirements is provided in this guide.”

A review that walks the five steps below and ends in a signed agreement can point at a named standard for the architecture and the regulatory-requirements language. That changes what the negotiation is about. Not whether a term is reasonable — whether this federation departs from a published framework, and why.

The evidence a federation owes its participants is site-level, not aggregate. And the poisoning drill in particular has a documented worst case to be tested against.

One compromised client, chosen once, is enough. Bagdasaryan and co-authors showed it in 2020, with a model-replacement attack: “Even a single-shot attack, where a single attacker is selected in a single round of training, causes the joint model to achieve 100% accuracy on the backdoor task.” The attack is cheap at scale too. In an 80,000-participant word-prediction task, 8 compromised participants reached 50% backdoor accuracy. Data poisoning needed 400 to do the same.

The same paper's abstract explains why the aggregation protocol is no help here. Federated learning “employs secure aggregation to protect confidentiality of participants' local models and thus cannot detect anomalies”. Confidentiality of the update and inspection of the update are one property read from two sides. So the drill is not one bad client slowly dragging an average. It is one client, one round, 100% on the attacker's task, and a rollback path that has to work after the fact.

Federated learning distributes the data and concentrates the governance question. So the agreement between institutions is the artifact to keep current.

What a current agreement looks like in production is a release rule with a budget attached to it. Google has more than twenty Gboard language models trained by federated learning, each carrying a ρ-zCDP guarantee for ρ ∈ (0.2, 2), two of them additionally trained with secure aggregation. The 2023 paper reporting that goes further than the twenty: “We are happy to announce that all the next word prediction neural network LMs in Gboard now have DP guarantees, and all future launches of Gboard neural network LMs will require DP guarantees.”

Read what that sentence commits to. It is not a description of twenty past models. It binds every future launch. It names the budgets rather than gesturing at privacy. And it puts secure aggregation on top of the guarantee for two of them, rather than in place of it. That is the shape of a rule a partner board can hold a coordinator to.

Name the site-level result, poisoning finding, or withdrawal request that would force the owner to redesign, restrict, remedy, or retire the system.