Graph Data Engineering and Dynamic Relations

Nodes and edges do not simply appear in nature. Product instrumentation, entity resolution, time windows, sampling, and policy decide which relations enter the graph, and those choices can dominate everything built on top of them.

The Metropolitan Police Service's Gangs Matrix is what an undeclared relation contract looks like at full scale. As of October 2017 nearly 4,000 people were on it. Amnesty International UK, investigating the database, reported that “78% are black, but in reality black people are responsible for just 27% of serious youth crime”. An entry could be generated by an indicator as weak as being a victim of gang violence, or the music videos a person had shared. The UK Information Commissioner's Office investigated in 2018 and reported the result plainly: “The ICO found that, while there was a valid purpose for the database - to tackle violent gang crime - the way it was being used across London was unlawful: it didn’t properly distinguish between victims of crime and offenders and was being shared more widely than was needed.” The Met later removed hundreds of people from it. Then it discontinued the database.

Stated as data engineering, the failure is specific. Two different relations — was harmed by gang violence, and participated in gang violence — were collapsed into one edge type. The observation process that produced the edges decides who gets noticed and whose shared videos get read. It was never written down beside them. The result was then distributed as though it were a set of facts about people, rather than a record of what one process had recorded about them.

The same ambiguity sits in every missing edge. An absence may mean no relationship, an unobserved relationship, a privacy rule that forbade recording it, or a relationship that had not yet formed. Graph data engineering therefore begins with relation semantics and temporal availability, not with a graph neural network library.

Each component needs identity, time, provenance, and versioning. None of these are theoretical artefacts. The Temporal Graph Benchmark ships a fixed, published negative set alongside its edges, so that an evaluation can be repeated. PinSage's paper states its sampling policy down to the rank interval its hard negatives are drawn from. A release that records only the nodes and the edges has thrown away the parts a reviewer would need to reconstruct the task.

A payment network links accounts, devices, merchants, addresses, and prior cases. The walkthrough below is a mechanism, not a report of one incident. It is how an investigation's own conclusions travel backwards into the training data.

The shape you store should match how relationships form and become available.

The third column rests on a distinction: when a relationship held in the world, against when your system could see that it held. That is not an exotic requirement invented for graph machine learning. It has been in the SQL standard since SQL:2011, which ISO/IEC published in December 2011. Two kinds of table are defined there. Application-time period tables carry validity periods that the user sets and may later correct. System-versioned tables carry SYSTEM_TIME period columns that are GENERATED ALWAYS AS ROW START and ROW END, and users cannot alter them, so that the recorded history “cannot be tampered with”. A table that declares both is bitemporal. Kulkarni and Michels, writing up the standard's temporal features for IBM, gave the reason for wanting both: “Such tables are very useful for capturing both the periods during which facts were believed to be true in the real world as well as the periods during which those facts were recorded in the database.”

An edge table carrying both periods can be queried AS OF a past instant. That is the as-of reconstruction in the third column below, expressed in the query language rather than reimplemented in application code. Implementations such as MariaDB support all three forms.

The process should preserve what the system knew before each prediction.

The Temporal Graph Benchmark is steps 3 to 6 carried out in public, on nine datasets that run up to tgbn-token's 72,936,998 edges. The cutoff is stated rather than implied: “All datasets are split chronologically into the training, validation, and test sets, respectively containing 70%, 15%, and 15% of all edges”. The negatives are designed rather than assumed. Each positive edge is ranked against multiple negatives drawn equally from historical and random negatives, and the negative set is fixed and published, so two teams evaluate against the same one. The benchmark also publishes a per-dataset surprise index, the fraction of test edges never seen in training. A model that is mostly memorising can then be recognised as one, rather than congratulated.

That the protocol is written down is what allows it to be checked. Le Yu re-ran nine methods on five TGB link datasets and three node datasets, under the same protocol and the same metrics. The rankings differ sharply from one dataset to the next.

Track node and edge counts by type, degree distributions, component sizes, isolated-node rates, identity-merge volume, relation latency, and sampling coverage. Sudden embedding changes may originate from a source outage, an aggressive entity merge, or altered candidate generation, rather than from model drift.

Some of these statistics also belong in the release itself, not only on a dashboard. TGB's surprise index — the fraction of test edges never seen in training — is a construction diagnostic published beside the data. It is what lets a reader see that a memorising baseline such as EdgeBank is scoring on repetition rather than on prediction. A metric that can only be computed from the graph, not from the model, is often the one that explains the model.

When the graph is used in multiple products, lineage should identify which release, sampling policy, and excluded relations informed each model.