TangledFeatures: Robust Feature Selection in Highly Correlated Spaces

AI summary

Overview

Research area: Machine learning methodology — feature selection for highly correlated predictor spaces, applied to structural biology (molecular conformation prediction).

Technical level: Intermediate. The paper assumes familiarity with correlation matrices, random forests, feature importance measures, and standard regression metrics, but each component is described in enough detail to follow without specialist background.

Scope: The paper introduces TangledFeatures, a three-stage pipeline (correlation clustering, stability-based representative selection, importance-based refinement), and evaluates it against five alternative feature selection approaches on the Alanine Dipeptide benchmark for predicting the backbone torsional angles phi and psi.

What This Paper Is About

Most feature selection methods optimize predictive accuracy and break down when predictors are strongly correlated with one another, producing unstable and redundant feature sets. The paper's goal is a selection framework that identifies one representative feature from each cluster of entangled predictors, so the final subset is compact, reproducible across resampled data, and still predictive. The authors test this on Alanine Dipeptide (Ace–Ala–Nme), where intra-atomic distances are used as inputs to predict the backbone torsional angles phi (C–N–C-alpha–C) and psi (N–C-alpha–C–N).

Key Contributions

1. TangledFeatures pipeline. A stability-based feature selection pipeline for correlated feature spaces, composed of a correlation module (c_alpha), a selection module (s_beta), and a refinement module (r_gamma).
2. Quantitative validation. Evaluation of predictive accuracy and feature stability against LASSO, Elastic Net Regularization (ENR), Random Forest Recursive Feature Evaluation (RFE), Boruta, and a no-selection baseline, across 24 pipelines in total.
3. Interpretability demonstration. Evidence that the retained distances map onto structurally meaningful intra-atomic distances in Alanine Dipeptide, including N–C, C–C, and C–N interactions previously established as determinants of phi and psi variability.
4. Software release. A reproducible R package, TangledFeatures, available on GitHub and prepared for CRAN submission, implementing redundancy detection, pruning, stability analysis, and correlation-structure visualization.

Main Findings

• Predictive accuracy stays competitive. All models achieve high predictive accuracy (R-squared greater than 0.9 for most pipelines). The best performance for phi came from SVM with Elastic Net (RMSE = 0.05, R-squared = 0.99), and the best for psi also came from SVM with Elastic Net (RMSE = 0.84, R-squared = 0.89).
• The accuracy trade-off is deliberate. The paper reports that SVM's higher accuracy arises from leveraging redundant features, whereas TangledFeatures trades a slight loss in accuracy for a more stable, non-redundant feature set.
• TangledFeatures accuracy figures. For phi, TangledFeatures reported OLS 0.26 / 0.87, RF 0.09 / 0.98, XGBoost 0.09 / 0.97, and SVM 0.09 / 0.98 (RMSE / R-squared). For psi, it reported OLS 0.97 / 0.61, RF 0.67 / 0.81, XGBoost 0.97 / 0.60, and SVM 0.86 / 0.75.
• Superior stability under the Kuncheva index. For the phi angle, TangledFeatures achieves consistently higher overlap of the top-k SHAP features across bootstrap runs than ENR or RFE. ENR shows sharp drops in overlap when correlated predictors are present, and RFE often retains different redundant distances, leading to low reproducibility. TangledFeatures curves remain flat and close to the maximum, indicating nearly identical feature subsets across resamples.
• Superior stability under Spearman rank correlation. ENR and RFE correlations fluctuate strongly across runs, reflecting instability in the relative importance assigned to correlated predictors. TangledFeatures achieves near-perfect rank correlation across resamples, showing that both the selected subset and the relative ordering of features are highly reproducible.
• Selected features are chemically meaningful. The retained distances consistently aligned with backbone and near-backbone interactions, such as N–C, C–C, and C–N, that are well-established determinants of phi and psi torsional variability.
• LASSO is compact but less interpretable. LASSO retained the second-fewest features among the baselines, but often selected redundant or chemically less meaningful distances, limiting interpretability despite its compactness.
• Selected distance lists. For phi, the retained distances include ACE1-CH3 to ALA2-CB (Cap-related/Flexible), ACE1-C to ALA2-C (Flexible), ACE1-CH3 to ALA2-CA (Flexible), ACE1-C to ACE1-O (Rigid), ACE1-CH3 to NME3-N (Cap-related), ACE1-C to NME3-N (Cap-related), and ACE1-CH3 to ALA2-C (Rigid). For psi, they include ACE1-CH3 to ALA2-C (Rigid), ACE1-C to NME3-N (Cap-related), ACE1-C to ACE1-O (Rigid), ACE1-C to ALA2-O (Rigid), ACE1-CH3 to NME3-N (Cap-related), and ACE1-C to ALA2-C (Flexible).
• Where the method is not the right tool. The conclusion states that where predictive accuracy is the sole objective and interpretability is less critical, PCA or LASSO may remain competitive alternatives. Classical methods may either oversimplify (for example PCA) or retain redundant features (for example LASSO and Boruta).

Methodology in Plain English

The pipeline runs in three stages, summarized by the paper as d → c_alpha → d_c → s_beta → d_s → r_gamma → d' → f(phi, psi).

1. Clustering. From a feature matrix D of n conformations by m intra-atomic distances, the method computes the pairwise Pearson correlation matrix Sigma. It then builds an undirected graph where nodes are distances and an edge connects two distances if the absolute value of their correlation is at or above a user-specified threshold tau. The connected components of this graph are the correlated clusters. To handle the periodicity of angles, the targets phi and psi are represented in cosine–sine form (cos phi, sin phi, cos psi, sin psi), which removes discontinuities at angular boundaries.
2. Selection. For each correlated cluster, the method picks a single representative distance. It trains multiple random forests; in each run, one candidate distance is sampled from each cluster and evaluated alongside all uncorrelated variables. Feature importance scores are averaged across runs, and the distance with the highest average importance in each cluster is retained. This ensemble approach is described as reducing variance and preventing arbitrary choices among correlated distances.
3. Refinement. The representatives are ranked by their Random Forest importance scores, and features are retained in descending order until cumulative importance reaches 0.99, so the final subset d' captures at least 99 percent of the predictive signal.

Evaluation. Each feature selection method was paired with four predictive algorithms — OLS, Random Forest Regression, XGBoost, and SVM — yielding 24 pipelines. Models were trained on 80 percent of the data and evaluated on the remaining 20 percent, with the whole training and evaluation process repeated 10 times using independent bootstrap resamples of the training data. Accuracy was measured by RMSE and R-squared. Stability was measured with SHAP values per run, Spearman rank correlation between runs, and the Kuncheva index, which compares the overlap of the top-k most important features across runs adjusted for chance.

Why This Matters

Correlated predictors are the norm rather than the exception in real datasets, and unstable feature selection undermines scientific reproducibility — the same analysis run twice can produce different conclusions. By placing feature stability at the center of the pipeline and evaluating it explicitly, TangledFeatures addresses a gap that accuracy-only benchmarks hide. The Alanine Dipeptide case study shows the payoff directly: the features the method selects correspond to real structural interactions, which is what makes downstream scientific claims actionable.

Real-world applications:

• Mutational studies and protein design, where identifying the residues and structural interactions that drive a conformation is a prerequisite for making targeted changes.
• Biomarker discovery in high-dimensional genomics, one of the extension domains the authors state is underway.
• Financial modeling in complex markets, also named by the authors as a planned extension domain.
• Large-scale government and public-sector datasets, the third extension domain named in the paper, where correlated administrative variables are common.

Industry relevance: The paper argues that post-hoc explanation methods such as SHAP, Integrated Gradients, and Global Importance Analysis produce explanations that are unstable in the presence of highly correlated structural features and may fail to yield biologically meaningful drivers. A selection method that returns a stable, compact, redundancy-aware subset is better suited to regulated or high-stakes settings where a model's stated drivers must survive re-analysis. The released R package and CRAN submission signal an intent to support practitioner adoption beyond the research benchmark.

Future Directions

• Extending beyond the Alanine Dipeptide proof-of-concept. The paper positions the Alanine Dipeptide study as a proof-of-concept; documented extensions underway cover high-dimensional genomics, complex financial markets, and large-scale government datasets.
• Establishing when accuracy or interpretability should dominate. The conclusion explicitly leaves open that PCA or LASSO may remain competitive when predictive accuracy is the sole objective, raising the question of a principled criterion for choosing between the two goals.
• Sharper treatment of correlated-space structure. The method currently depends on a user-specified correlation threshold tau to form clusters; the paper does not report sensitivity analysis of that choice, leaving open how robust the clusters are to threshold selection.
• Reproducibility of the approach itself. The authors frame the contribution partly as a tool for reproducible research and released the pipeline as an open package, but independent replications on other correlated systems are not reported in this paper.

Target Audience

Researchers and practitioners who need feature selection that remains stable under correlated predictors — particularly computational structural biologists, cheminformatics and drug-discovery scientists, and machine learning engineers working with high-dimensional collinear data such as genomics, finance, or administrative records. It is also relevant to methodologists interested in stability selection and to reviewers or analysts who need to justify why a given feature subset should be trusted across repeated analyses.

Note: the paper does not report the number of conformations n, the number of intra-atomic distances m, the specific correlation threshold tau used, the number of random forest runs, or tree counts.

Authors’ abstract

Feature selection is a fundamental step in model development, shaping both predictive performance and interpretability. Yet, most widely used methods focus on predictive accuracy, and their performance degrades in the presence of correlated predictors. To address this gap, we introduce TangledFeatures, a framework for feature selection in correlated feature spaces. It identifies representative features from groups of entangled predictors, reducing redundancy while retaining explanatory power. The resulting feature subset can be directly applied in downstream models, offering a more interpretable and stable basis for analysis compared to traditional selection techniques. We demonstrate the effectiveness of TangledFeatures on Alanine Dipeptide, applying it to the prediction of backbone torsional angles and show that the selected features correspond to structurally meaningful intra-atomic distances that explain variation in these angles.