Computing patient similarity based on unstructured clinical notes

AI summary

Overview

Research area: Clinical natural language processing and electronic health record (EHR) mining, applied to machine learning for precision medicine.

Technical level: Intermediate. The paper assumes familiarity with text embeddings (LSA, Doc2Vec, transformer models), matrix similarity measures, and rank-correlation evaluation, but it explains each component clearly enough for a reader with basic machine learning background.

One-sentence scope: The paper proposes and clinically validates a modular pipeline that represents each patient as a matrix of embeddings of their unstructured clinical notes and computes patient-to-patient similarity from those matrices, tested on 4,267 Czech breast-cancer patients with expert annotations.

What This Paper Is About

Electronic health records contain rich clinical detail, but most of it is locked in free-text notes rather than structured fields, which makes large-scale analysis difficult. Instead of trying to extract structured facts from the text, the authors take a complementary approach: they represent an entire patient by the embeddings of all their notes stacked into a matrix, and then measure how similar two patients are by comparing their matrices. The goal is to produce similarity scores that reflect distinct clinical facets (such as treatment or family history) and that could later support personalized therapy recommendations or toxicity warnings.

Key Contributions

1. A method for computing patient similarity directly from clinical notes, where notes are automatically segmented and filtered by a technique from prior work, then converted into patient matrices.
2. A modular experimental pipeline that lets any state-of-the-art text embedding method be combined with any matrix similarity technique, with source code released on GitHub (https://github.com/ZepZep/patient-similarity).
3. A validation methodology for assessing computed similarities across ten clinical contexts, using annotations from three clinicians at the Masaryk Memorial Cancer Institute.
4. A detailed empirical analysis of 42 parameter combinations, including a discussion of which similarity facets the system can and cannot capture, and why.

Main Findings

• Filtering by clinical category helps: Restricting notes to segments relevant to a given similarity category improved performance mainly for the LSA and combined vectorization techniques, but not for Doc2Vec.
• RV coefficient similarity (Rrv2) is the most consistent matrix method: It worked reasonably well with all vectorization techniques, whereas the MaxMax (Rmms) and Edit-distance (Reds) methods were more sensitive to the choice of embedding.
• Best-performing configurations reach a mean Kendall rank correlation of 0.31: The top-scoring combinations were Rrv2 with Vlsa050 (filtered), Rrv2 with combined (filtered), and Rrv2 with Vlsa200 (filtered), all at 0.31 mean correlation across the ten categories.
• The combined ensemble stabilizes results: Ensembling all three vectorization methods at dimension 50 improved performance substantially for the Reds matrix method specifically.
• Doc2Vec underperformed: Especially in combination with Rmms or Reds; the authors attribute the poor showing of Vd2v200 with filtering to insufficient training data for Doc2Vec at dimension 200 once irrelevant segments are discarded.
• Only some similarity facets are captured: Six of the ten categories (Age, Medical history, Social history, Medication, Allergies, Side effects) are described as mostly random predictions. Type of tumor, Treatment, and Treatment type perform reasonably well, with Family history also partially captured.
• Annotation reliability is uneven: Inter-annotator agreement was poor for Social history, Allergies, and Side effects, making results for those categories unreliable.
• Age cannot be recovered from the notes: Age information is simply not present in the clinical note texts.
• Runtime grows sharply with embedding dimension for Rrv2: On a machine with 12 CPU cores, computing the full matrix for 4,267 patients (approximately 9 million pairwise scores) took 3.5 minutes with Rrv2 at dimension 50 but 280 minutes at dimension 200, whereas Rmms took 15 and 50 minutes, and Reds took 140 and 400 minutes.
• Finetuning was the one impactful hyperparameter: Most vectorization hyperparameters had minimal effect, with the notable exception of finetuning the RobeCzech transformer model, which greatly improved performance.

Methodology in Plain English

The authors built a three-stage pipeline. First, each clinical note is split into segments (paragraphs), and a BERT-based model is trained to predict section titles for the roughly 40% of segments that lack them, since about 60% already contain titles such as "Medication:" or "Summary:". Using a vector space of titles, the system semi-automatically decides which segment types are relevant to which of ten similarity categories, then discards irrelevant segments. Second, each remaining segment is converted into a vector using one of three techniques: Latent Semantic Analysis, Doc2Vec with the PV-DM algorithm, or a RobeCzech transformer [CLS] embedding compressed with SVD. Stacking these vectors vertically gives one matrix per patient, with as many rows as that patient has notes and as many columns as the embedding dimension. Third, each pair of matrices is reduced to a single similarity number using either the RV coefficient (a multivariate extension of Pearson correlation adapted by Smilde et al. to handle differing numbers of notes), a MaxMax approach based on row- and column-wise maximum cosine similarities, or an edit-distance method similar to dynamic time warping that finds the best time-consistent path through the pairwise cosine-similarity matrix.

Evaluation used a grid of 42 combinations: two filtering options (on and off), seven vectorization options (LSA, Doc2Vec, and RobeCzech at dimensions 50 and 200, plus a combined ensemble at dimension 50), and three matrix similarity methods. The ground truth came from a validation study in which ten pivot patients deemed representative were each paired with five relevant patients selected by a baseline system (two similar, one neutral, two dissimilar, scored using LSA embeddings and cosine similarity over concatenated records). Three clinicians rated each pair on a 0–10 scale across the ten categories, with −1 allowed for incomparable pairs, yielding 500 scores per annotator. Results were measured with the Kendall rank correlation coefficient because it handles small samples and tied values more robustly than Spearman's rho.

Why This Matters

Impact on research: The work reframes the EHR-mining problem by showing that patient-level similarity can be computed without first solving the hard problem of comprehensive structured information extraction from clinical text. It also provides an honest account of which clinical facets text-based similarity can and cannot capture, and how validation design constrains the strength of conclusions in this area.

Real-world applications:

• Personalized therapy recommendations based on the treatment histories of clinically similar patients.
• Alerts about likely treatment toxicity, drawing on the side-effect profiles of similar patients.
• Survival or treatment-response prediction informed by the clinical trajectories of comparable patients.
• Cohort identification and clinical decision support within hospital information systems.

Industry relevance: The pipeline is modular and runs on commodity hardware, and the published code plus runtime measurements give practical guidance for anyone integrating patient similarity into hospital or research software. The finding that Rrv2 slows down dramatically at higher dimensions (3.5 versus 280 minutes) is directly relevant to deployment decisions. The authors also note that their results could feed downstream tasks such as personalized therapy recommendations or toxicity warnings, though no deployment or production system is reported in the paper.

Future Directions

1. Enlarge the validation study: The authors want more pivot patients and more than five comparison patients per pivot, which would make the study statistically robust and allow splitting data into train and test parts for supervised machine learning approaches to patient similarity.
2. Refine the similarity category definitions: Several categories had low inter-annotator agreement, and the authors plan to clarify their clinical specifications to improve agreement, and to prepare a more focused benchmark dataset with clinicians.
3. Improve filtering precision: Being stricter about which segment titles are relevant to each category could reduce noise from short, superficially similar notes, at the cost of leaving some patients with fewer or no segments.
4. Address the low-note-count categories: Categories such as Age, Family history, Social history, Allergies, and Side effects currently suffer from few relevant segments and high sensitivity to individual notes; the paper identifies this as a target for nearer-term work.

Target Audience

Clinical informatics researchers and machine learning practitioners working on EHR text mining, patient similarity, or clinical decision support will find the pipeline design and the parameter grid most useful. Clinical researchers and oncologists involved in retrospective study design will benefit from the validation methodology and the candid discussion of annotation agreement and category definition problems. Data scientists evaluating text embedding and matrix comparison techniques for small, domain-specific text corpora in a non-English language will also find the empirical comparisons directly applicable.

Authors’ abstract

Clinical notes hold rich yet unstructured details about diagnoses, treatments, and outcomes that are vital to precision medicine but hard to exploit at scale. We introduce a method that represents each patient as a matrix built from aggregated embeddings of all their notes, enabling robust patient similarity computation based on their latent low-rank representations. Using clinical notes of 4,267 Czech breast-cancer patients and expert similarity labels from Masaryk Memorial Cancer Institute, we evaluate several matrix-based similarity measures and analyze their strengths and limitations across different similarity facets, such as clinical history, treatment, and adverse events. The results demonstrate the usefulness of the presented method for downstream tasks, such as personalized therapy recommendations or toxicity warnings.