Identifiers, Keys, and Entity Resolution

Entity resolution fails in two directions. A false merge welds two subjects into one. A false split fragments one subject into many. The first has been counted.

ECRI Institute PSO collected wrong-patient reports for 32 months, from January 2013 through August 2015. It filtered 10,915 collected reports down to 7,613 wrong-patient events, submitted by 181 healthcare organisations, and coded 7,740 failure modes in them. Its August 2016 Deep Dive on patient identification reports that 91.4% were caught before harm reached the patient. The remainder is what an identity error costs.

One of the sample events reads: “A patient in cardiac arrest was mistakenly not resuscitated because the care team pulled up the wrong patient’s record and adhered to a do-not-resuscitate order.” The U.S. Government Accountability Office repeated the case independently in January 2019, citing ECRI's Deep Dive. A false merge is not a row-count anomaly. It is one record's instructions executed on another person.

A single identifier rarely serves every join safely. The loss from crossing a scope boundary has been measured with the identifier, the vendor and the organisation all held fixed. Kaiser Permanente runs 17 instances of Epic across its regions. A 2014 report for the Office of the National Coordinator for Health Information Technology records what that costs: “For example, Kaiser Permanente (which has 17 instances of Epic across its regions) reported a match rate of greater than 90 percent within each instance; that rate fell to around 50 percent to 60 percent when sharing between regions using a separate instance of Epic or with outside Epic partners.” Thirty to forty points of match rate disappear because the namespace changed, not because the patients did.

The same report looked at master patient indexes. Studies of 112 MPIs gave a mean duplication rate of eight percent, and 11 combined EMPIs ranged from seven to 39 percent duplication. The Government Accountability Office found the same order of magnitude in 2019: “A 2014 study found that as few as 50 percent of records are accurately matched when organizations exchange information”. Where a key is designed to survive cross-scope use, the scope is written into the standard instead of assumed. The Global Legal Entity Identifier Foundation states that “Every LEI is unique and can only ever represent one entity”.

Joining tables decides which records describe the same subject, which events belong to one episode, and which context is attached to an example. Those are substantive assumptions about the world. SQL will happily execute a many-to-many join that multiplies rows. The result may contain valid-looking values, pass type checks, and still change the learning population dramatically.

Keys also have lifecycles. Accounts merge, devices reset, email addresses are shared, and external identifiers can be recycled. A field that is unique today may not be globally or historically stable. The gap is measurable rather than theoretical: the same organisation, the same EHR vendor and the same patients produce a match rate above 90 percent inside one Epic instance and around 50 to 60 percent across instances.

Building a dataset reliably begins by declaring key scope, join cardinality, temporal validity, and duplicate policy before computing features.

Use assertions at every join boundary rather than checking only the final row count, and treat step 5 as a question about who is missing rather than how many.

Match failure is not spread evenly across people. A 2016 Census Bureau report by Joshua Comenetz counts the names: “About 6.3 million different surnames were reported by census respondents in 2010”, covering 295 million people, of which “3.9 million, or 61.9 percent, were reported only once”. Collision is where the groups separate. The ten most common surnames cover 4.9 percent of the total population, and it takes 239 surnames to reach a quarter of it. For the White alone population the figures are 4.5 percent and 319 surnames. For Black alone, 13.0 percent and 43. For Asian and NHPI alone, 13.4 percent and 41. And then: “The highest level of clustering is found in the Hispanic population, where just 26 surnames cover a quarter of the population and 16.3 percent of people reported one of the top 10 names.” A surname-based blocking key therefore collides far more often in one group than in another, before any algorithm has been chosen.

The Brennan Center for Justice describes the consequence in a deployed system: “It often flagged false positive matches because it relied on a combination of name matches, which inaccurately picked out common names, and date of birth matches, which were frequently missing or inaccurately entered into state systems”.

A single reconciliation figure hides all of that. The Census Bureau's 86 percent assignment rate on the 2010 Decennial file is where step 5 begins, not where it ends. The question is which slices make up the remaining fourteen percent, and whether they are the same slices next month.

Entity resolution changes as new evidence arrives. A merge made today should not silently rewrite which records were linked at an earlier scoring cutoff.

Version merges, splits, aliases, and confidence so historical datasets can reconstruct the identity graph available at the time, and monitor the model impact of large reconciliation changes.

The Financial Stability Board wrote that contract into a live global scheme. Its 8 June 2012 report to the G20 sets out 35 recommendations for a global Legal Entity Identifier. One of them fixes uniqueness: “only one LEI may be assigned to any financial market counterparty”. Another fixes the shape of the key, against the standard ISO 17442:2012, dated 30 May 2012: “The technical code of the LEI should be a 20 digit alpha numeric number as set out in the ISO standard 17442:2012. The code should not incorporate any intentional embedded intelligence (such as a country reference) which would lead to the code becoming out of date.” A surrogate key, one issuing authority, and no meaning buried inside the string.

The temporal half of the contract comes in a footnote: “For entities with a date of expiry, the reason for the expiry should be recorded, and, if applicable, the LEI of the entity or entities that acquired the expired entity.” An expiry date, a reason, and a pointer to the successor is precisely the record a merge should leave behind. The old identity is not overwritten. It is closed and linked, so a dataset built last year can still be reconstructed. The Global Legal Entity Identifier Foundation, which now operates the Global LEI System, describes the resulting code as “a unique 20-character alphanumeric code that enables anyone, anywhere in the world, to access clear, unique identification data about a legal entity”.