Learning Event Instrumentation and Taxonomy

Each missing field removes a class of future questions. Or it creates a biased proxy.

Feature engineering can transform observed data. It cannot recover context the source never captured. If a recommendation log stores clicks without impressions, the team cannot tell what users ignored. Instrumentation defines the evidence available to future models. Event names, identifiers, timestamps, payloads, and logging conditions determine which behavior becomes visible. A useful event represents one occurrence with stable semantics. It records who or what acted, what happened, which object was involved, the relevant context, and when the occurrence took place.

What a missing context field costs was measured, not assumed. In 2005 a Cornell team sat subjects in front of a search engine with an eye tracker running. Phase I recruited 34 subjects, with usable eye-movement data for 29 of them. Phase II recruited 22, with usable data for 16 — 6 “normal”, 5 “swapped”, 5 “reversed”. Subjects viewed the abstracts at ranks 1 and 2 almost equally often. They clicked rank 1 far more. Thorsten Joachims and four colleagues put it this way: “It is very interesting that users click substantially more often on the first than on the second link, while they view the corresponding abstract with almost equal frequency.” In the normal condition, the more relevant of the top two results was clicked in 19 of 20 single-click cases when it was placed first, and in only 2 of 7 when it was placed second. The authors call the effect trust bias. Microsoft Research reported the same dependence independently three years later, in 2008, its abstract stating that “the probability of click is influenced by a document's position in the results page”. The same click, in the same log, means different things depending on what was displayed beside it. The impression list, not the click, is the field that makes the click interpretable.

Logging also creates risk. Over-collection increases privacy and security exposure. Aggressive sampling can remove rare cases or distort propensities. The first of those has a public price. From 2014 to 2019, more than 140 million users gave Twitter phone numbers and email addresses for account-security purposes, and the data was then used for targeted advertising. On 25 May 2022 the FTC and the Department of Justice sued. FTC Chair Lina M. Khan, in the press release announcing the charges: “As the complaint notes, Twitter obtained data from users on the pretext of harnessing it for security purposes but then ended up also using the data to target users with ads.” Twitter agreed to a $150 million civil penalty, and the stipulated order was signed on 26 May 2022. The design must balance learning value with minimization and operational cost.

These components make the record interpretable after it leaves the application that created it. Three of them are already specified in published work rather than left to each team's house style.

Identity is standardised, sized and validated. W3C Trace Context became a W3C Recommendation on 23 November 2021, and it fixes the correlation identifier that travels between systems. The traceparent header carries a 1-byte version, a 16-byte trace-id, an 8-byte parent-id (span-id) and 8 bits of trace-flags. All-zero values are invalid, and vendors are required to ignore a traceparent whose trace-id is invalid. Under the heading “Uniqueness of trace-id”, the requirement is one line: “The value of trace-id SHOULD be globally unique.” OpenTelemetry, a separate body, binds implementations to the same header — “A W3C Trace Context propagator MUST parse and validate the traceparent and tracestate HTTP headers”. That shared identifier is what lets a command emitted by one service and the result recorded by another be joined afterwards.

Time is an interval, and the systems that take it seriously publish the width. Google described TrueTime in the Spanner paper in 2013. The API returns an interval rather than a point, and the paper states the width: “In our production environment, ϵ is typically a sawtooth function of time, varying from about 1 to 7 ms over each poll interval.” That bound is roughly 4 ms most of the time. It is built out of a 30-second daemon poll interval, an applied drift rate of 200 microseconds per second (0 to 6 ms), and about 1 ms of communication delay to the time masters. The Sundial paper, in 2020, cites the same order of magnitude from outside Google, at “<10ms in TrueTime”. The move worth copying is not the hardware. It is publishing the uncertainty next to the timestamp instead of recording the timestamp as though the uncertainty were zero.

Governance metadata carries purpose, and purpose is the field the Twitter penalty attached to. The $150 million was not for collecting phone numbers. It was for reusing security data as an advertising join key. Consent, purpose, retention class and sensitivity have to travel with the value, because the record outlives the reason it was collected.

The record type determines which questions can be reconstructed later.

“Can be duplicated or late” is not a caveat about events. It is a design assumption with a published answer. Apache Kafka's baseline delivery guarantee is at-least-once, so producer retries put duplicates in the log. KIP-98 does not remove the retries. It gives every record a stable identity — a producer id (PID) plus a per-partition sequence number that starts at zero and increases monotonically — and lets the receiver do the checking. From the “Idempotent Producer Guarantees” section of KIP-98: “The broker will reject a produce request if its sequence number is not exactly one greater than the last committed message from that PID/TopicPartition pair.” A duplicate then becomes an expected, non-fatal condition, returned as an error the producer can ignore. A gap in the sequence is the fatal one. Confluent reports that the mechanism shipped in Kafka 0.11 with “a negligible impact on producer throughput”.

The same principle joins a command to its outcome: a shared identifier carried across the hop, not a match on timestamps.

Treat instrumentation as a versioned production interface rather than something analytics bolts on informally.

Step 2 is the one teams improvise, and it is the one that has already been designed twice in public. Kafka's KIP-98 defines identity as a pair: a producer id and a per-partition sequence number that starts at zero and increases monotonically. Deduplication becomes a check the receiver performs rather than a promise the sender makes. W3C Trace Context defines the join key that crosses system boundaries: a 16-byte trace-id, all-zero values invalid, required to be globally unique, with an 8-byte parent-id naming the step inside the trace. Between them they cover both jobs a stable ID has to do. Recognise the same event arriving twice. Recognise the same activity observed in two places. Copy the shape rather than inventing a convention that only this pipeline understands.

Track event volume by application version, platform, geography, experiment, and key entity slices, because a stable total can hide a complete outage for one segment.

England ran that failure at national scale. Between 25 September and 2 October 2020, 15,841 positive COVID-19 test results were silently dropped from the data load into the NHS Test and Trace contact-tracing system. No component reported an error. Individuals still received their own results. The transfer files were simply too large. Public Health England said so in a statement on 4 October 2020: “The technical issue was caused by the fact that some files containing positive test results exceeded the maximum file size that takes these data files and loads then into central systems.” The loss was discovered overnight on Friday 2 October, eight days after it began. Over 75% of the missing cases — 11,968 of them — should have been reported in the final three days of the window, when contact tracing would still have been timely. A peer-reviewed study by Fetzer and Graeber attributes to that episode more than 125,000 additional infections and over 1,500 additional COVID-19 deaths. The only visible symptom for eight days was a volume shortfall nobody was watching for.

Measure duplicate IDs, invalid timestamps, payload size, missing required context, clock skew, and end-to-end delay. Compare client and server events when both represent the same action. Clock skew has a known scale to argue from. If a fleet with dedicated time masters and a 30-second poll interval carries an error bound of about 1 to 7 ms, the timestamp a mobile client writes deserves a wider bound, stated rather than assumed.

Create synthetic canary events that should traverse the entire path. Canaries distinguish “no user activity” from a broken collection system. They are precisely the control that would have turned England's eight days of silence into an alert on the first one.

Finally, preserve raw records long enough to investigate transformations, subject to governance limits. Early normalization should not erase the evidence needed to diagnose the source.