Privacy, Personalization Boundaries, and Data Governance

A displayed slate can reveal an interest to whoever is looking at the screen. So can an explanation, an ad, or a profile control. On the system side the risk spreads just as wide: raw events, derived features, embeddings, model memorization, cross-device linking, provider reports, and household sharing.

Data minimization begins with the recommendation purpose. Collect, retain and expose a signal only when its incremental value is proportionate and its lifecycle can be governed. That is not only a house rule. The GDPR's Article 5(1)(b) requires collection for specified, explicit and legitimate purposes. Article 5(1)(c) limits the data to what is necessary for those purposes.

Every row of that risk list has a published demonstration behind it. Ratings carrying no names were re-identified from a public film database in 2008. Membership in a training set was recovered from a black-box commercial service at 94% median accuracy in 2017. Training images were reconstructed from shared gradients alone in 2019. In January 2021 the Federal Trade Commission ordered a company to delete not only the photos but the models built from them. This lesson works from those demonstrations, not from hypotheticals.

Data does not stay where it was collected. It becomes profiles, embeddings and inferred traits. Those feed ranking, experiments and provider analytics. By the time someone asks for deletion, the original event is the easiest of the five rows to remove.

GoodRx Holdings Inc. shared its users' prescription medications and health conditions with Facebook, Google, Criteo, Branch and Twilio. It then used that data to target those same users with medication-specific ads. On 1 February 2023 the Federal Trade Commission announced its first-ever enforcement action under the Health Breach Notification Rule. The FTC's press release describes one of those transfers: “For example, in August 2019, GoodRx compiled lists of its users who had purchased particular medications such as those used to treat heart disease and blood pressure, and uploaded their email addresses, phone numbers, and mobile advertising IDs to Facebook so it could identify their profiles.”

The proposed order carried a $1.5 million civil penalty. The Department of Justice filed it on the FTC's behalf, as United States v. GoodRx Holdings, Inc., and the case was terminated on 17 February 2023. Nothing here required a model to be wrong. The inference was accurate and the identifiers matched. That is precisely what made the transfer worth an enforcement action.

Purpose creep is gradual and technique theatre is instant. Signals gathered for one surface drift into durable cross-product profiling over years. Federated learning or synthetic data gets called automatically private in a single slide.

There is a measurement under that second claim. Shared gradients alone were enough to rebuild the training data. In 2019 a paper called Deep Leakage from Gradients recovered pixel-wise accurate training images and token-wise matching text from nothing else. Its first section states the shape of the failure: “In centralized distributed training (Fig. 1a), the parameter server, which usually does not store any training data, is able to steal local training data of all participants.” The system that holds no data reads the data.

Its defences come with thresholds rather than assurances. Gaussian or Laplacian noise stopped the attack only once the variance exceeded 10^-2. At that level “noise with variance larger than 10-2 will degrade the accuracy significantly” — CIFAR-100 accuracy falling from 76.1% to about 45-46%. Gradient compression stopped it once the pruning ratio went above roughly 20% sparsity, since “the maximum tolerance of sparsity is around 20%”. Int-8 quantization also prevented the leakage, at an accuracy of 53.7%. Both half-precision formats, IEEE float16 and bfloat16, failed to protect the training data at all. A technique is private at a setting, not by name.

Verify that opting out changes the recommendation policy, that deleting history removes derived influence, and that temporary mode does not update durable state. Conduct inference and membership-risk testing where appropriate. It is worth knowing what such a test returns.

Models trained through Google's and Amazon's prediction services gave up who was in their training data. The default configurations were enough. On 10,000-record retail transaction datasets, an attacker guessed membership at a median accuracy of 94% against the Google-trained models and 74% against the Amazon-trained ones. The 2017 paper reports it in the introduction: “For multi-class classification models trained on 10,000-record retail transaction datasets using Google's and Amazon's services in default configurations, our membership inference achieves median accuracy of 94% and 74%, respectively.” Fully synthetic shadow data still reached 90% against the Google-trained models. Against a model trained on the Texas hospital discharge dataset — 67,330 records, 6,170 binary features — the attack exceeded 70%. The question is not whether the training set is visible. It is how often an attacker guesses membership correctly, against a black-box service nobody had to misconfigure.

Track performance losses from privacy controls across user groups. A privacy mechanism that disproportionately degrades sparse or minority profiles may have to be redesigned rather than simply accepted. That degradation has been measured. On an MNIST class holding only 500 examples, a non-private model reached 97% accuracy on “8” against 99% on “2”. The differentially private model reached 77% on “8” against 98% on “2”. In a controlled run the well-represented classes fell from 99% to 98% and the underrepresented class fell from 93% to 77%. The 2019 paper that reports this opens its abstract with the finding: “We demonstrate that in the neural networks trained using differentially private stochastic gradient descent (DP-SGD), this cost is not borne equally: accuracy of DP models drops much more for the underrepresented classes and subgroups.” An average would have reported a small cost.

Every signal is tied to a named recommendation decision. Or it is not collected. That first step is not a house rule invented for this lesson. The GDPR's Article 5(1)(b) requires that personal data be collected for specified, explicit and legitimate purposes. Article 5(1)(c) requires that it be limited to what is necessary for those purposes. The European Data Protection Board restates the same test for design work: “Only personal data that is adequate, relevant and limited to what is necessary for the purpose shall be processed.”

Step 3 asks that events, profiles, embeddings and indexes expire coherently. That is the rule the FTC enforced at Everalbum, where the face recognition data and the models built from the photos were ordered deleted alongside the photos themselves. Disclosure and deletion are then tested at the end, across shared devices, sensitive topics, data exports and cache invalidation. Tested, because a deletion path that has never been run against a cache is an assumption about a cache.