Extrapolation of Periodic Functions Using Binary Encoding of Continuous Numerical Values

AI summary

Overview

Research area: Machine learning, specifically input data representation and representation learning for neural networks.

Technical level: Intermediate. The paper assumes familiarity with multi-layer perceptrons, activation functions, Fourier features, and training concepts such as regularization and learning rate schedules, but its central idea (binary encoding of numbers) is simple enough to explain without deep mathematics.

One-sentence scope: The paper reports that encoding continuous numerical inputs in binary form (Normalized Base-2 Encoding, NB2E) lets ordinary multi-layer perceptrons extrapolate periodic functions beyond the range they were trained on, where fixed Fourier encoding and raw continuous inputs fail.

What This Paper Is About

Neural networks generally cannot predict a periodic signal outside the range of values they were trained on, even when they fit the training data well. The authors show that this limitation depends heavily on how the input number is represented: converting a normalized value into its binary digits, rather than feeding the raw number or sinusoidal features, lets a plain MLP continue a periodic signal into unseen territory without any knowledge of the function's mathematical form.

Key Contributions

1. Introduces Normalized Base-2 Encoding (NB2E) as a method for encoding continuous numerical values, in which a value in [0, 1) is represented as a vector of N bits following Equation 3, with each bit position i carrying positional value 2^-i.

2. Demonstrates extrapolation by a vanilla MLP. Using NB2E input, a simple five-layer, 512-neuron-per-layer MLP extrapolates several periodic functions beyond the training domain, while Fixed Fourier Encoding (FFE) with identical frequencies and continuous numerical input both fail at the task.

3. Establishes the conditions under which NB2E works. The training data must span the 0.5 boundary and, in practice, reach a normalized maximum of at least 0.7, so that every element of the binary vector takes on both 0 and 1 during training (Equation 4). A normalization value z < 2·max(X) (Equation 5) controls the extrapolation horizon.

4. Explains the mechanism through activation analysis. UMAP projections of hidden-layer activations, followed by DBSCAN clustering, reveal that NB2E induces "bit-phase" internal representations in which the network organizes information by signal phase and specific bit patterns rather than by position.

Main Findings

• NB2E beats both alternatives on a simple sine. For f(x) = sin(x) with 10,000 points drawn i.i.d. from U(0, 100) and a roughly 70/30 ordinal train/test split at X'_train ≤ 0.7 < X'_test < 1.0, NB2E made the most accurate predictions, FFE predicted well immediately past the training domain but drifted in phase and amplitude, and continuous numerical input failed catastrophically, unable to fit even the training data.

• NB2E extends to composite sines. For f(x) = sin(x) + 2.5·sin(3.7 + 1.4x) with 10,000 points from U(0, 400), NB2E both learned the function and maintained accuracy in extrapolation. FFE fit the training data nicely but failed to extrapolate. Continuous input could not even fit the training data.

• NB2E handles non-sinusoidal composites. For the sawtooth/triangle composite f(x) = 2(x/3.1 − ⌊x/3.1 + 0.5⌋) + 4|x/5 − ⌊x/5 + 0.5⌋| − 1 with 10,000 points from U(0, 200), NB2E learned and extrapolated better than the other methods, with occasional large residuals. FFE's extrapolation only vaguely resembled the signal; continuous input again failed.

• Performance degrades gracefully with added complexity. For the three-part composite f(x) = 2(sin(2πx/2.1) + sin(2πx/2.3)) + 2exp(−9.7(x/7.2 − ⌊x/7.2⌋)) + 0.7·sgn(sin(2πx/12.2)), residuals increased but remained substantially smaller than those of the other methods.

• Training must cross the 0.5 boundary. In a split-point sweep from p = 0.1 to p = 0.7 in increments of 0.1, prediction failed for every p ≤ 0.5, because the first NB2E bit (value 0.5) was never activated during training. Failure was not catastrophic but appeared as a phase shift: outputs past the bit transition restart as if from x' = 0.

• Bit transition points are the failure locations. For p = 0.1, prediction failed progressively at x' = 0.125, x' = 0.25, and x' = 0.5, corresponding to the third, second, and first NB2E bits. Even p = 0.6, which satisfies Equation 4, failed at the transition x' = 0.75 and showed poor fitting at x' = 0.25, with a smaller error at x' = 0.6875 = 1/2 + 1/8 + 1/16.

• Sufficient numbers of cycles are needed. Varying the number of complete sine cycles from 1 to 7 while holding the split and point counts fixed, one or two cycles were insufficient to learn anything resembling the sine curve; 3–5 cycles began to capture periodicity and range with residual jumps; 7 cycles reached near-perfection. Additional cycles improved refinement but were visually indistinguishable.

• Small datasets work surprisingly well. Sweeping the dataset size from 250 to 2,000 points in increments of 250, the prediction still resembled a sine curve with only 250 points. Sensitivity to data size was highly dependent on the data itself, but the authors report it was practical even with small datasets in most cases.

• Noise is tolerated. Adding zero-mean Gaussian noise with standard deviation from 0.25 to 2.0 in increments of 0.25 to the sine example, the MLP recovered the sine curve, with extrapolation quality decreasing as noise increased.

• Activations organize by phase, not position. UMAP projections of the 512-D activations to 3-D followed by DBSCAN clustering showed groupings distinguished primarily by phase of the sine curve, with each phase-group spanning much of the positional space. Clustered groups shared a distinctive signature in bits 5, 6, and 7, which near-uniquely represented signal phase. Because the network never saw any NB2E of a point with x' > 0.7, the authors interpret the result as projection of a learned bit-phase space onto position rather than positional extrapolation.

• Composite signals produce dynamic, multi-phase representations. For the composite sine function, Period 1 (P = 2π) and Period 2 (P = 2π/1.4 ≈ 4.49) were represented jointly. Layer 1 was spherical and organized by Period 1 phase; layer 2 split into two groups separated perfectly by bit 7, with the two periods represented orthogonally; layer 5 yielded four groups each containing two distinct bit patterns for bits 4 and 7. Period 1 phase distributions were nearly identical for groups 1/4 and 2/3 and Gaussian-like; Period 2 distributions were broader, with groups 1/3 and 2/4 similar.

• Sinusoidal activations help. Replacing ELU with sine activations while changing nothing else noticeably improved predictions for the first three examples, with little effect on the fourth. The authors conclude sine is the superior activation for NB2E.

• NB2E is claimed as a first. The authors state that, to their knowledge, NB2E is the first method enabling extrapolation of unknown periodic functions through input encoding alone, without architectural modifications or prior knowledge of the functional form, distinguishing it from symbolic regression, physics-informed neural networks, and Neural ODEs.

Methodology in Plain English

The idea is to stop handing a network a raw number and instead hand it the number's binary digits. Every input value is first rescaled into the range [0, 1). Then that value is written as a sum of binary fractions, and the resulting string of ones and zeros—48 of them in this work—becomes the network's input vector. Each position, or "bit," has a fixed meaning: the first bit is worth 0.5, the second 0.25, the third 0.125, and so on.

The comparison is deliberately controlled. The authors pit NB2E against two alternatives: Fixed Fourier Encoding, which uses sine and cosine pairs at frequencies 2^(i−1) (so 96 inputs, since each frequency needs two terms), and plain continuous numerical input (a single value). Table 1 shows that NB2E bit i corresponds to FFE at frequency 2^(i−1), so the two encodings cover the same hierarchy of scales but with discrete steps versus continuous waves.

The network itself is intentionally ordinary: five dense layers of 512 neurons each with ELU activation, L2 regularization at a factor of 10^-4 on each hidden layer, and a final single linear neuron. No periodic activation functions were used in the main experiments, precisely so that the encoding could be tested on its own rather than letting sinusoidal activations supply implicit structure. Training used the AdamW optimizer with a cosine annealing schedule with warm restarts, mean absolute error as the objective, and 4,000 epochs in every case. Batch size was 1000, and training took 3–5 minutes on a single NVIDIA Tesla V100 GPU.

The tests move from simple to complex: a single sine, a two-sine composite, a sawtooth/triangle composite, and a three-part composite mixing sines, an exponential, and a sign function. Inputs were drawn randomly, with 10,000 points in each case, to rule out learning from regular sampling. Splits were ordinal, not random, placing the training data first in the domain so the test region represents genuine extrapolation.

Beyond the headline comparisons, the authors ran sensitivity studies on the split point (0.1 through 0.7), the number of signal cycles (1 through 7), the number of data points (250 through 2,000), and the level of added Gaussian noise (standard deviation 0.25 through 2.0). To understand why the method works, they projected each hidden layer's 512-dimensional activations down to three dimensions with UMAP and clustered them with DBSCAN, coloring the results by phase, by position, and by cluster.

Why This Matters

The paper argues that a network's ability to extrapolate can come from how numbers are represented rather than from architecture, added physics, or discovered equations. The authors also note two practical properties of NB2E predictions: they are equally valid throughout the extrapolation domain, unlike sequence models that accumulate error step by step, and they are continuous, so predictions can be made at any representable value rather than only at fixed sampling intervals.

Real-world applications discussed or implied by the paper:

• Irregular time series. Because NB2E predictions can be made for any representable value below unity, the authors suggest it suits data that is not regularly sampled, unlike sequence models that learn dependencies on order rather than on the actual axis value.
• Representation of time. The paper states that NB2E grew out of work on representing time, and describes it as a "magnitude-free representation of magnitude" that could serve as an input feature in sequence prediction models where time's numerical value matters to the output.
• Astrophysical time-series analysis. The authors cite prior astrophysical work (Kostov et al., 2025; Powell et al., 2021) on representing time series as 1D shapes, a practice they note also discards the numerical value of time.
• Signals where the underlying function is unknown. Since NB2E requires no prior knowledge of functional form and no architectural change, it is presented as applicable wherever periodic behavior must be continued forward without an available equation.

Industry relevance. The work targets any setting where a periodic quantity must be forecast from past observations alone—sensor streams, seasonal demand, orbital or rotational signals, and similar phenomena. The low cost matters here: the network has five layers, training runs in 3–5 minutes on a single NVIDIA Tesla V100 GPU, and the authors report useful results from as few as 250 data points in their smallest test.

Future Directions

• Refining the optional combination with periodic activations. The authors describe their test of sine activations with NB2E as minimal, having run no new sensitivity or activation analysis, and state they suspect additional benefit from fine-tuning the MLP or improving its structure alongside sinusoidal activation.

• Applying NB2E to sequence prediction. The paper speculates that the NB2E representation of time could be powerful as an input feature in sequence prediction models where the value of time relates to the output, but does not test this.

• Explaining why bit 7 and specific bits carry the decisive information. The activation analysis identifies bits 5, 6, and 7 as near-uniquely representing phase in the sine case and bits 4 and 7 in the composite case, and notes that DBSCAN could no longer be trusted to identify all independent groups in the later, interleaved layers of the composite signal.

• Finding the true complexity ceiling. The authors say they kept complicating the function "searching for a level of complexity where the NB2E MLP would no longer be viable," but the final example still succeeded, so the limit was not reached. They also report a lack of a systematic sensitivity comparison across examples beyond the sine curve for dataset size.

Target Audience

This paper is best suited to machine learning researchers and practitioners working on coordinate-based networks, implicit neural representations, time-series modeling, or representation learning—particularly those familiar with Fourier feature encodings and looking for an alternative input scheme. It is also relevant to domain scientists who must extrapolate periodic signals without knowing the governing equation. Readers need only an intermediate grasp of neural network training; the novel concept itself is elementary, since it amounts to encoding numbers as their binary digits.

Authors’ abstract

We report the discovery that binary encoding allows neural networks to extrapolate periodic functions beyond their training bounds. We introduce Normalized Base-2 Encoding (NB2E) as a method for encoding continuous numerical values and demonstrate that, using this input encoding, vanilla multi-layer perceptrons (MLP) successfully extrapolate diverse periodic signals without prior knowledge of their functional form. Internal activation analysis reveals that NB2E induces bit-phase representations, enabling MLPs to learn and extrapolate signal structure independently of position.