Tensor Network Based Feature Learning Model

AI summary

Overview

Research area: Machine learning — kernel methods, tensor networks, and hyperparameter optimization for high-dimensional regression.

Technical level: Advanced. The paper assumes fluency with kernel machines, tensor decompositions (Canonical Polyadic Decomposition, Kronecker products, Khatri-Rao and Hadamard products), Fourier features, and alternating least squares optimization.

Scope: The paper introduces a Feature Learning (FL) model that replaces cross-validation over Fourier-feature hyperparameters with a jointly learned linear combination of tensor-product feature maps, trained by an ALS algorithm on a CPD-structured parameter tensor.

Note: the version of the paper supplied here is truncated; the text ends partway through Section 4.1. Sections after that (including any conclusion) are not available, so this summary covers the abstract, background, method, and the reported numerical tables only.

What This Paper Is About

Kernel methods are accurate but require computing a kernel matrix at a cost of at least O(N³), which is impractical for datasets with N ≈ 10⁷–10⁹ samples. One fix is to work in the primal formulation and map data into a high-dimensional tensor-product feature space, then constrain the model parameters w to a tensor network format (a rank-R CPD), which shrinks storage from O(I^D) to O(DIR) and cost per gradient step to O(DIR).

That solves the parameter side, but the feature hyperparameter θ (the periodicity of the Fourier basis) is still chosen by generic cross-validation, which ignores the model's structure. The FL model instead treats a set of P candidate θ values as a learnable rank-P CPD, so the importance weights λ_p are learned from data alongside w, in a single optimization problem rather than P separate ones.

Key Contributions

1. A new Feature Learning model. The paper replaces a fixed feature map φ_θ with the learnable combination f(x) = [Σ_{p=1}^{P} λ_p φ_{θ_p}(x)]ᵀ w, where the feature map in brackets is itself a rank-P CPD and λ_p serves as a measure of the importance of hyperparameter θ_p.

2. An ALS training algorithm. By fixing λ to update the CPD cores W^(d) and fixing w to update λ, the method splits one non-linear problem into a sequence of smaller linear problems. Training complexity drops from O(I^{2D}[N + I^D]) to O(ℰDNIR[P + IR]), with peak memory O(NR[P + I]).

3. Three regularization alternatives for λ. The paper derives and compares L1 (proximal gradient), L2 (closed-form ridge solution), and Fixed Norm (‖λ‖₂² ≤ 1, solved via SVD and a scalar non-linear equation), each with complexity O(NP²), and also tests non-negativity-constrained variants.

4. A quantized implementation and empirical comparison. The implementation uses quantized Fourier features (I_d = 2^{K_d}, each ψ_θ^(d) written as a Kronecker product of vectors in ℂ²), reducing the algorithm to O(ℰDNKR[P + R]) time and O(NRP) memory. Results are benchmarked against a quantized CPD kernel machine with 6-fold cross-validation.

Main Findings

• Speedup claim: The paper reports that the FL model can be consistently trained 3–5 times faster than a standard cross-validated model while achieving prediction quality on par with it.

• Airfoil (N=1502, D=5, I=4, R=51): FL MSE 0.184 ± 0.02 vs CV MSE 0.223 ± 0.02; FL time 3.0 s vs CV time 23 s.

• Energy (N=768, D=8, I=4, R=15): Both methods reach MSE 0.003 ± 0.0; FL time 0.91 s vs CV time 5.5 s.

• Yacht (N=308, D=6, I=2, R=6): FL MSE 0.112 ± 0.02 vs CV MSE 0.358 ± 0.06; FL time 0.149 s vs CV 0.615 s.

• Concrete (N=1030, D=8, I=8, R=10): FL MSE 0.139 ± 0.03 vs CV MSE 0.118 ± 0.02; FL time 1.2 s vs CV 8.8 s. This is the one reported dataset where the CV baseline has a lower mean MSE.

• Wine (N=6497, D=11, I=16, R=25): FL MSE 0.692 ± 0.07 vs CV MSE 0.652 ± 0.04; FL time 33 s vs CV 152 s.

• Airline (N=5,929,413, D=8, I=64, R=20): FL MSE 0.804 ± 0.0 vs CV MSE 0.779 ± 0.0; FL time 15,159 s vs CV 56,590 s. This is the large-scale case, run with P=6 and 5 restarts rather than P=8 and 10 restarts.

• Regularization comparison: Across Airfoil, Concrete, Energy, Wine, and Yacht, the six regularization settings produced comparable MSE and training time. The L1 and L1(P) settings achieved slightly better MSE values and trained faster on average. The paper attributes this to sparsity: some λ_p become exactly zero, so fewer contributions are needed when building the A_d matrix.

• Complexity reductions proven analytically: Model responses and gradients of the CPD kernel machine are computable in O(DIR) instead of O(∏_{d=1}^{D} I_d). A single core update costs O(N[I_d R P + I_d² R²]) when N ≫ I_d R, with memory O(N I_d R).

• CPD chosen over alternatives: The paper argues CPD has only one hyperparameter R, whereas Tensor Train has D−1 rank hyperparameters, and CPD avoids the exponential dependence on D present in the Tucker decomposition.

• Fourier features chosen over polynomial features: Polynomial features are noted to exhibit instabilities during training due to ill-conditioned Vandermonde matrices.

Methodology in Plain English

The starting point is a standard way of handling non-linear regression without kernel matrices: push each input through a tensor-product feature map and do linear regression in that large space. The problem is that the feature space has I₁I₂…I_D dimensions, so the weight vector w would be astronomically large. The trick is to store w as a Canonical Polyadic Decomposition — a sum of R Kronecker products of small vectors — which makes storage and computation scale linearly in the data dimension D.

The leftover problem is the feature hyperparameter θ. Normally you would try several values, train a separate model for each, and pick the best by cross-validation, which means solving the whole optimization problem P times. The FL model instead writes the feature map as a weighted sum of P different feature maps, and treats the weights λ_p like any other model parameter. Because that sum is itself a rank-P CPD, everything keeps the efficient tensor structure.

Training alternates between two steps. First, with λ frozen, each CPD core W^(d) is updated in turn by solving a regularized least squares problem in closed form; a precomputing trick keeps intermediate matrices Z_p and H up to date so each core update is cheap. Second, with w frozen, the λ vector is updated from the matrix F = [Φ₁w … Φ_P w] using one of three regularization schemes: L1 via proximal gradient steps, L2 via a closed-form ridge solve, or a fixed-norm constraint solved through an SVD. This loop runs for ℰ epochs. The released implementation uses quantized features, which replaces each I_d × R core with a chain of 2 × R cores.

Experiments use 5 UCI regression datasets plus the Airline dataset, with an 80/20 train/test split, inputs scaled to the unit hypercube, outputs standardized, I_d = I for all d, and R and I chosen so the parameter count stays below the sample size (an underparameterized regime intended to prevent memorization). The baseline is a quantized CPD kernel machine with 6-fold cross-validation over θ ∈ {10, 2, 128, 25, 64, 600, 2000, 1024}. Training runs 10 epochs and is repeated 10 times (5 for Airline); reported numbers are mean and standard deviation of test MSE plus training time in seconds. All experiments except the large-scale one ran on a Dell Inc. Latitude 7440 laptop with a 13th Gen Intel Core i7-1365U CPU and 16 GB of RAM. Code and data are on GitHub at https://github.com/AlbMLpy/TN-FL-Model.

Why This Matters

Impact on research. Hyperparameter search is often treated as an external wrapper around a model rather than part of it. This paper shows that when the feature map has a tensor-product structure, the hyperparameter selection can be folded into the CPD itself and optimized jointly. That reframes cross-validation as an optimization sub-problem and gives tensor-network kernel machines a complexity advantage they previously lacked on the feature side, not just the parameter side.

Real-world applications (framed by the benchmark datasets used, not by deployment claims in the paper):

• Tabular regression on engineering design data such as the Airfoil dataset, where the model must handle modest sample sizes with several input dimensions.
• Materials and structural property prediction, represented by the Concrete dataset, where a small model that trains in about a second is valuable for rapid iteration.
• Energy system modeling, represented by the Energy dataset, where both methods matched at MSE 0.003 but the FL model trained roughly six times faster.
• Large-scale industrial regression, represented by the Airline dataset with 5,929,413 samples, where reducing training from 56,590 s to 15,159 s changes what is feasible to retrain.

Industry relevance. The main practical lever is training cost on datasets where cross-validation over feature hyperparameters would otherwise multiply the expense by P. The reported Airline result — the largest dataset tested, at N = 5,929,413 — is the clearest case, though the paper also reports that the CV baseline had a slightly lower mean MSE on Airline (0.779 vs 0.804) and on Concrete (0.118 vs 0.139), so the speedup comes with a small accuracy trade-off on some data.

Future Directions

• Reducing the accuracy gap on some datasets. The FL model trails the cross-validated baseline in mean MSE on Concrete and Airline. The paper's reported content does not provide a resolution of this gap.

• Extending beyond Fourier features. The paper notes that polynomial features are avoided due to ill-conditioned Vandermonde matrices, but states that other generic functions can be used for the high-dimensional mapping. Which alternatives behave well under the FL formulation is left open.

• Choosing the number of candidate hyperparameters P and the set of θ values. The experiments use P = 8 for most datasets and P = 6 for Airline, with the θ candidate list fixed as {10, 2, 128, 25, 64, 600, 2000, 1024}. How P and that list should be selected in general is not addressed in the available text.

• Tuning the remaining hyperparameters. The Fixed Norm regularization was introduced specifically to eliminate the β hyperparameter from the λ sub-problem, but α, R, I, and ℰ remain. Whether these can also be learned rather than set is an open question the paper raises without answering.

Target Audience

Researchers and practitioners working on scalable kernel methods, tensor-network machine learning, and high-dimensional regression will get the most from this paper. It is also relevant to anyone who routinely runs cross-validation over feature hyperparameters on large tabular datasets and wants a structurally informed alternative. Readers need a working knowledge of tensor decompositions and regularized least squares to follow the derivations; the numerical results in Tables 1 and 2 are readable without that background.

Authors’ abstract

Many approximations were suggested to circumvent the cubic complexity of kernel-based algorithms, allowing their application to large-scale datasets. One strategy is to consider the primal formulation of the learning problem by mapping the data to a higher-dimensional space using tensor-product structured polynomial and Fourier features. The curse of dimensionality due to these tensor-product features was effectively solved by a tensor network reparameterization of the model parameters. However, another important aspect of model training - identifying optimal feature hyperparameters - has not been addressed and is typically handled using the standard cross-validation approach. In this paper, we introduce the Feature Learning (FL) model, which addresses this issue by representing tensor-product features as a learnable Canonical Polyadic Decomposition (CPD). By leveraging this CPD structure, we efficiently learn the hyperparameters associated with different features alongside the model parameters using an Alternating Least Squares (ALS) optimization method. We prove the effectiveness of the FL model through experiments on real data of various dimensionality and scale. The results show that the FL model can be consistently trained 3-5 times faster than and have the prediction quality on par with a standard cross-validated model.