A Common Measure of Communication for Speech Brain-Computer Interfaces

AI summary

Overview

Research area: Speech brain–computer interfaces (BCIs) and machine-learning evaluation methodology, with an information-theoretic framing.

Technical level: Intermediate. The paper is written accessibly, but the core argument relies on entropy and mutual information, so some familiarity with information theory helps.

Scope: This paper derives open-vocabulary mutual information (OVMI), an information-theoretic measure that places speech BCI systems evaluated under different datasets, tasks, modalities, and vocabularies onto a single, explicitly defined communication scale.

What This Paper Is About

Speech BCIs turn neural activity into language, but the field has no common way to measure progress because systems differ in datasets, recording methods, speech task, and vocabulary size, so reported scores are rarely comparable. The authors argue that ordinary accuracy and word error rate (WER) are computed only over the words a system supports, which can overstate how much of a user's intended speech the system can actually convey. Their goal is to define a measure that answers two questions: what distribution of words a speech BCI should let a user communicate, and how much information from that distribution the system can convey.

Key Contributions

1. Diagnosis of conventional metrics. The paper shows why accuracy, WER, in-vocabulary entropy, and Wolpaw's information transfer rate (ITR) can overstate communication, and decomposes the overstatement into a coverage gap and a non-uniformity (uniform-prior) term.

2. Derivation of OVMI. The authors derive open-vocabulary mutual information, defined as lexical coverage C(S) multiplied by the mutual information between intended and decoded words conditional on the intended word being in the decoder vocabulary, together with a general confusion-matrix estimator, a scalar-accuracy estimator, and a per-word variant.

3. Cross-system comparison on a common scale. Using a single reference distribution, they compare heterogeneous speech BCI studies — invasive and non-invasive, attempted and perceived speech — and expose trade-offs between how much language a system supports and how accurately it decodes those words.

4. OVMI as a vocabulary-design objective. They show that selecting vocabularies to maximise OVMI improves held-out word accuracy, with peak relative improvements over frequency-based selection of 15.4%, 16.3%, and 8.4% across three speech domains (up to 16.3% cited in the abstract).

Main Findings

• In-vocabulary measures overestimate open communication. For a noiseless decoder whose vocabulary is the top-V most frequent words under the reference distribution p, Wolpaw's uniform-prior quantity gives log2 V bits, accounting for actual word frequencies gives H(p_S), and OVMI gives C(S)H(p_S), with C(S)H(p_S) ≤ H(p_S) ≤ log2 V. The gap is largest for small vocabularies, where lexical coverage C(S) is far below 1.

• A toy example reverses the ranking. With 1,000 equally likely intended words, System A (vocabulary 50, 100% in-vocabulary accuracy, 5% lexical coverage) scores 0.28 bits under OVMI, while System B (vocabulary 1,000, 50% accuracy, 100% coverage) scores 3.98 bits — accuracy ranks A above B, OVMI ranks B above A.

• Information is lost to either coverage or fidelity, and the pattern differs by modality. The 50-word invasive systems convey comparatively high in-vocabulary information but cover only a small fraction of broad spoken English. Most evaluated non-invasive systems achieve greater coverage through frequency-selected vocabularies but substantially lower in-vocabulary information. The later 125k-word invasive systems combine near-complete coverage with high decoding fidelity.

• Measured progress is dominated by vocabulary expansion, not fidelity gains. On the normalised OVMI scale, the 2021 50-word Moses system conveys 2.4%, rising to 4.7% with language-model post-processing. The 50-word Willett results in 2023 remain in the same regime at 6.7% and 6.4%. Willett's 125k-word system reaches 72.0% and Card reaches 93.7% in 2024 — the largest historical increase occurs at the transition to large-vocabulary decoding, after in-vocabulary fidelity had already approached saturation at smaller vocabulary sizes.

• Non-invasive word decoding sits in a lower information regime. The d'Ascoli et al. (2025) decoder yields 2.4% on LibriBrain100 and 1.8% on Armeni, MEG-XL yields 0.3% on MEG-MASC, and Tang's fMRI system reaches 3.6%. The authors note the particularly low MEG-MASC value occurs with little data per participant across a large, heterogeneous subject population.

• Comparisons depend on what the user is expected to communicate. Against four reference distributions (SUBTLEX-UK for broad spoken English, Switchboard for conversation, AAC/UCV, and Sherlock Holmes for narrative speech), large-vocabulary systems are largely insensitive: Card conveys 93.7–97.5% and the 125k-word Willett system 72.0–76.2%. Small-vocabulary systems are much more sensitive: the 50-word Willett system conveys 6.4% of broad spoken-English information but 40.4% under the AAC reference, and Moses rises from 4.7% to 30.7%.

• System rankings can flip with the reference distribution. LibriBrain100 and the isolated-word Moses system are nearly matched under broad speech (2.43% versus 2.4%), but Moses leads under AAC (16.5% versus 7.34%), while LibriBrain100 leads under narrative speech (2.62% versus 1.0%).

• OVMI matches or beats alternative vocabulary-selection strategies. Using a contrastive decoder trained once on a 250-word candidate lexicon, selecting the inference vocabulary by OVMI matched or exceeded frequency selection, validation-accuracy selection, and random selection across vocabulary sizes, with the advantage diminishing at larger vocabulary sizes (at V = 250 all methods converge because the candidate pool contains 250 words).

• OVMI generalises existing measures. When C(S) = 1, OVMI reduces to ordinary mutual information within the decoder vocabulary; if intended words are also uniform and errors symmetric, it reduces to the per-trial quantity underlying Wolpaw's ITR.

Methodology in Plain English

The authors begin by defining a reference distribution p over words — an explicit statement of what a user might wish to communicate. They use SUBTLEX-UK, derived from film and television subtitles, as broad spoken English, and additionally test Switchboard, an AAC/UCV list, and Sherlock Holmes.

They then define lexical coverage C(S) as the probability that a word drawn from p falls inside the decoder's vocabulary S. If the intended word is outside S, they assume the user abstains and the output is empty. Under this model, mutual information decomposes into a binary-entropy term indicating whether the word is supported, plus a term measuring information conveyed among supported words. They keep only the second term and call it OVMI.

Because most published studies report only a scalar accuracy or WER and a vocabulary size, the authors use a Wolpaw-like scalar estimator: each supported word is decoded correctly with probability P, and errors spread uniformly over the other V−1 words. They instantiate P as macro accuracy (the uniform average of per-word correct-decoding probabilities), reasoning that non-uniform word frequency should enter through the external reference distribution rather than through the evaluation set's frequency weighting.

For the empirical comparison, they take reported vocabularies and accuracies or WERs from published systems and compute OVMI against a shared reference, normalising by the entropy of that reference to give percentages. For the vocabulary-selection experiment, they train the d'Ascoli et al. (2025) contrastive decoder once on a 250-word candidate lexicon, then restrict inference retrieval to subsets of varying size chosen by frequency, validation accuracy, OVMI, or random selection, repeating across three communication domains and five seeds.

Why This Matters

Impact on research. Benchmarks compare methods within a fixed experimental setting but cannot relate capabilities across settings. OVMI complements benchmarks by mapping heterogeneous results onto one communication objective, which is useful retrospectively for studies outside any benchmark and prospectively if benchmarks saturate or are superseded. The authors recommend reporting OVMI alongside standard decoder metrics. They also provide an OVMI explorer and a Python package at the project's GitHub pages.

Real-world applications:

• Clinical communication for people with paralysis — comparing intracortical attempted-speech decoders against each other on a scale tied to how much of a user's intended speech is actually conveyed, not just how accurately supported words are decoded.
• Assistive and augmentative communication (AAC) design — the AAC/UCV reference shows that vocabularies designed around caregiving communication cover far more of that restricted distribution than of unrestricted speech, informing how such devices are specified.
• Vocabulary curation for deployed decoders — treating OVMI as a selection objective rather than picking the most frequent words.
• Cross-modality technology assessment — judging whether invasive and non-invasive approaches are progressing toward the same communication target, while respecting that perceived-speech decoding does not by itself imply a usable interface.

Industry relevance. The coverage-versus-fidelity trade-off and the finding that gains came primarily from vocabulary expansion rather than decoding improvements are directly relevant to neurotechnology developers, and more broadly to any constrained-vocabulary speech system where reported accuracy is conditional on the supported word set. The paper's citation of information transfer rate in recent work (including Neuralink Corporation, 2026) shows these metrics are in active industrial use.

Future Directions

• Extending OVMI beyond lexical information. The authors state they evaluate lexical rather than contextual information, and leave extension to a conditional language distribution to future work.
• Paraphrase-aware measurement. OVMI treats out-of-vocabulary words as unsupported, even though meaning could sometimes be expressed through paraphrasing; the authors chose to keep the measure tied to explicitly available lexical information and flag this as a limitation.
• Better estimators from richer reported statistics. The retrospective comparison relies on the scalar estimator because confusion matrices are rarely reported; when confusion matrices or per-word accuracies are available, OVMI can be computed directly, and the authors encourage this.
• User-centred evaluation. The paper places user-centred evaluation as a distinct, higher-cost question ("Is the system practically useful?") beyond what OVMI answers, and recommends it before practical deployment.

Target Audience

Speech BCI and neuroengineering researchers who need to compare decoders built under incompatible conditions; machine-learning researchers working on evaluation metrics for constrained-vocabulary or open-vocabulary decoding; clinicians and assistive-technology designers choosing vocabularies for communication devices; and industry teams in neurotechnology or speech interfaces who report accuracy or WER as headline numbers and want a more principled way to characterise what those numbers mean.

Authors’ abstract

Speech brain-computer interfaces (speech BCIs) translate neural activity into language, offering a path towards restoring speech for people with paralysis and, more broadly, enabling new forms of natural human-computer interaction. Despite this promise, the field lacks a common measure of progress because systems use different datasets, recording methods, types of speech, and vocabularies, so their reported scores are rarely comparable. Underlying this measurement problem are two unresolved questions: (i) what distribution of words should a speech BCI enable a user to communicate, and (ii) how much information from this distribution can a system convey. We address both by deriving open-vocabulary mutual information (OVMI), an information-theoretic quantity that measures the information conveyed by a decoder relative to a reference distribution over the words a user may wish to communicate. This allows capabilities measured under different conditions, such as distinct vocabularies, to be evaluated on a common communication scale. We show that ordinarily reported accuracy, word error rate (WER), and other metrics computed only over the words a system supports can overstate how much of a user's intended speech the system can communicate. We then use OVMI to compare existing systems, expose trade-offs between how much of the user's language a system supports and how accurately it decodes those words, show that these comparisons depend on what the user is expected to communicate, and demonstrate that selecting a vocabulary to maximise OVMI yields up to 16.3% relative improvement in accuracy across three speech domains. OVMI therefore provides the speech BCI community with a principled way to compare heterogeneous systems, improve vocabulary design, and measure progress in the field.