A Theory of the Mechanics of Information: Generalization Through Measurement of Uncertainty (Learning is Measuring)

AI summary

Overview

• Research area: Machine learning theory and methods, specifically a model-free, information-theoretic framework for inference based on surprisal (self-information).
• Technical level: Advanced. The paper develops formal definitions, maximum-entropy assumptions, and equations for surprisal, deviation, and case-influence probabilities, though it opens with an intentionally accessible intuitive summary.
• Scope (one sentence): The paper lays out the mathematical foundations of a framework that performs prediction, generation, causal discovery, anomaly detection, reinforcement learning, and time series forecasting directly from data by measuring uncertainty, and reports that this is the first comprehensive treatment of a research effort that started in 2011.
• Authors and venue: Christopher J. Hazard, Michael Resnick, Jacob Beel, Jack Xia, Cade Mack, Dominic Glennie, Matthew Fulp, David Maze, Andrew Bassett, and Martin Koistinen, all affiliated with Howso Incorporated. arXiv:2510.22809v1 [cs.LG], 26 Oct 2025, licensed CC BY-SA 4.0.

What This Paper Is About

Traditional machine learning requires building explicit models of data distributions, choosing loss functions, tuning hyperparameters, and making domain-specific assumptions, which limits flexibility and interpretability. This paper proposes instead that learning should be treated as a process of measuring and characterizing uncertainty in the relationships among data, rather than optimizing a specific objective function. The goal is a unified system that is practically usable across nearly every task, where every step traces back to the data with units of measurement and uncertainty along the way, and where "there is no model besides the data and its uncertainties."

Key Contributions

1. A model-free inference framework built on surprisal. Inference is performed by querying the data itself for relevant records, with relevance quantified by information-theoretic uncertainty rather than by a fitted model. The paper positions this in relation to k-Nearest Neighbors, case-based reasoning, and instance-based learning, and argues it improves on prior nearest-neighbor work using uncertainty (Mason et al., 2021; Agarwal et al., 2016) by using surprisal as the uncertainty measure.

2. Substitutability deviation as a new uncertainty measure. The authors introduce substitutability deviation (often "deviation"), which reflects how likely one data element can be replaced by another without losing predictive or informative value. It is described as neither strictly epistemic nor aleatoric but a combination of both given the data, and is bounded below by measurement error and above by the residual.

3. A derivable link between distance, uncertainty, and surprisal. Using the Łukaszyk-Karmowski metric with Laplace-distributed uncertainty, the paper shows the surprisal of one value given another reduces to the absolute difference divided by the deviation, giving a mean-absolute-error-based "distance" that carries uncertainty with it. Surprisal is proven to be a distance metric when deviations are constant, and approximately a metric locally when deviations are smooth.

4. An operational, editable data system. Because the framework is a database, features and records can be added, edited, removed, or excluded from queries; records can be left out and predicted instead of using traditional cross-validation; duplicated records can be merged by combining their probability masses (case weights); and feature contributions to a prediction can be measured by comparing predictions with and without features, without model-derived bias.

Main Findings

• Surprisal becomes simple arithmetic: The paper derives that the surprisal of observing one value in place of another equals the absolute difference between the values divided by the deviation (Equation 2), extending to a marginal surprisal formula for continuous features (Equation 6).

• A constant baseline surprisal of 1.5 nats: The base surprisal of a case compared to itself is 1.5 nats, which must be subtracted to obtain marginal surprisal. A worked coin example is given: assuming a 1 in 6000 chance of an American nickel landing on its edge (Murray and Teare, 1993), the surprisal of heads is 1.00024 bits, of edge is 12.55075 bits, and the marginal surprisal of the edge over the normal baseline is 11.55051 bits.

• Negligible effect within uncertainty: Using the expected-difference metric reduces the distinctiveness of differences that fall within the uncertainty, but the effect on differences far greater than the uncertainty is described as insignificant.

• Relation to neural architectures: Equation 14, which multiplies surprisal by a feature's probability of influence, is said to closely resemble the conjunction of probabilities of independent distributions underlying the no-flattening theorem of Lin et al. (2017), suggesting a relationship between this framework and hierarchical neural network behavior.

• Surprisal as a drop-in distance: Since surprisal behaves as a metric (globally with constant deviations, locally with smooth deviations), the authors state it can replace distance in distance-based methods beyond kNN, naming HDBSCAN (McInnes et al., 2017) and UMAP (McInnes et al., 2018).

• Defaults for influential cases: The set of influential cases is bounded by k_min to k_max, with defaults k_min = 1, k_max = ∞, a marginal probability threshold t = 1/e³, and "similar cases" defined by a fixed k.

• Causal asymmetry detection: By measuring how the uncertainty in one variable changes when another is included in prediction, the system can detect asymmetries in both uncertainty and predictability, revealing likely causal relationships and confounders, in contrast to correlational techniques that only find two-way associations. Time series features (lagged values and rates) are said to strengthen causal results by assessing whether one change precedes another.

• Anomaly detection mechanism: Anomalies are identified as records whose probability of being informative to their neighbors is much less than their neighbors' probabilities of being informative to each other.

• Claimed performance, but no benchmarks in the provided content: The abstract states the approach achieves "at or near state-of-the-art performance across most common machine learning tasks," and the paper says appendices contain empirical evidence. However, the truncated content provided here does not include any specific benchmark names, dataset sizes, baselines, or numeric result tables, so no empirical results can be reported from this excerpt.

Methodology in Plain English

The approach starts from a simple intuition: for any question you ask of data, most data points have essentially zero probability of being informative, and a few have a meaningful probability. If you can estimate those probabilities, you can find the relevant data, compute a best estimate, and characterize the uncertainty and distribution around it.

To estimate uncertainty, the system uses each feature — including the feature itself — to find relevant records, makes a prediction, and measures the average absolute difference between prediction and actual value. The authors describe this deliberate self-inclusion as an intentional overfit that acts as a proxy for features not present in the data. These uncertainties become per-feature deviations, which are then combined into probabilities that one record is informative for another.

The mathematics relies on mean absolute error (rather than standard deviation) as the measure of risk and uncertainty, following Taleb (2007) on robustness to outliers and interpretability. Because maximum entropy distributions make the fewest assumptions given local data, the absolute residuals are treated as exponentially distributed and the differences between values as Laplace distributed (which is two back-to-back exponentials). Cumulative residual entropy (Rao et al., 2004) is used to derive surprisal from the exponential distribution's cumulative density function. Case-level surprisal is the sum of per-feature surprisals, and exponentiating its negative yields the probability that one case is informative for another.

On top of these probabilities, the framework defines prediction (continuous values as probability-weighted averages over influential cases; nominal values as the mode of accumulated probability masses), generation by drawing from the distributions, conditioning by specifying constraint values, reinforcement learning by selecting the most informative records that maximize reward and drawing from that distribution, and data reduction by merging records whose probability masses can be combined. Weights are available throughout so a merged record can stand in for repeated occurrences.

Why This Matters

• Impact on research: The paper argues that unifying machine learning around measuring uncertainty may open a path to scalable machine learning and AI that is an alternative to neural networks while preserving human understandability of the underlying mechanics. It reframes learning as managing data rather than managing models, and states that its mathematical foundations constitute a "physics" of information.

• Real-world applications (as described in the paper):

  • Anomaly detection: flagging records whose informative probability toward neighbors is unusually low.
  • Causal discovery: identifying asymmetric influences and confounders rather than two-way correlations.
  • Time series forecasting and hypothetical modeling: via derived lag and rate features feeding the same machinery.
  • Generative inference and reinforcement learning: producing outputs by drawing from probability distributions, and selecting records that maximize reward.

• Industry relevance: The framework treats the dataset as an editable database, so training data can be directly edited or deleted and updates are efficient, which matters for data governance, traceability, auditability, and regulatory contexts where decisions must be traced back to their inputs. The authors release the software openly (github.com/howsoai, including the python module howso-engine-py and example notebooks in howso-engine-recipes) and refer to the implementation as Howso or the Howso Engine. The framework is also described as robust to outliers and able to handle missing data, and to scale by combining sufficiently duplicative records.

Future Directions

• Establishing empirical standing against established methods: The paper claims at or near state-of-the-art performance across most common machine learning tasks and points to appendices of empirical evidence, but the specific benchmark comparisons are not present in the provided content. A natural next step is wider independent comparison to neural network baselines.
• Extending beyond the assumed distributions: The derivations assume Laplace (and, for positive reals, exponential) uncertainty as maximum entropy distributions; the paper notes that solutions for other distributions could be used if appropriate, leaving that as open work.
• Scaling and further development of the data-reduction path: The paper frames automatic combination of duplicate records as the mechanism for larger data sets, implying open questions about how far this scales and how well it preserves inference quality.
• Broadening the connection to neural network theory: The observed resemblance between the framework's probability conjunction and the no-flattening theorem (Lin et al., 2017) is presented as a possible bridge to understanding how hierarchical neural architectures behave, inviting follow-up work.

Target Audience

Readers best served by this paper include machine learning researchers and theorists interested in information-theoretic and non-neural inference, practitioners building interpretable or audit-friendly prediction systems, and data scientists working with problems such as causal discovery, anomaly detection, and time series forecasting who need traceability from decision back to source data. Some familiarity with probability, information theory (surprisal, entropy), and nearest-neighbor methods is needed for the formal sections; readers who only want the intuition can follow the paper's dedicated intuitive summary.

Authors’ abstract

Traditional machine learning relies on explicit models and domain assumptions, limiting flexibility and interpretability. We introduce a model-free framework using surprisal (information theoretic uncertainty) to directly analyze and perform inferences from raw data, eliminating distribution modeling, reducing bias, and enabling efficient updates including direct edits and deletion of training data. By quantifying relevance through uncertainty, the approach enables generalizable inference across tasks including generative inference, causal discovery, anomaly detection, and time series forecasting. It emphasizes traceability, interpretability, and data-driven decision making, offering a unified, human-understandable framework for machine learning, and achieves at or near state-of-the-art performance across most common machine learning tasks. The mathematical foundations create a ``physics'' of information, which enable these techniques to apply effectively to a wide variety of complex data types, including missing data. Empirical results indicate that this may be a viable alternative path to neural networks with regard to scalable machine learning and artificial intelligence that can maintain human understandability of the underlying mechanics.