A Multi-level Analysis of Factors Associated with Student Performance: A Machine Learning Approach to the SAEB Microdata

AI summary

Overview

Research area: Educational Data Mining (EDM) and Explainable AI applied to large-scale national assessment data from Brazilian basic education.

Technical level: Intermediate. The methods (tree-based ensemble classifiers, SHAP, Boruta feature selection) are standard in applied machine learning, but the paper assumes familiarity with classification metrics and multi-source data integration.

Scope: A multi-level machine learning study that fuses student, teacher, school, and principal microdata from the SAEB 2023 assessment to classify 9th-grade and high school students as performing above or below average, and to explain which factors drive that prediction.

What This Paper Is About

Brazil's SAEB assessment collects questionnaire data from students, teachers, schools, and principals, producing one of the country's richest sources of information about basic education. The authors ask whether machine learning can combine these four data sources into a single model that predicts student proficiency, and more importantly, whether the model can reveal which factors matter most. The goal is not just accurate prediction, but interpretable evidence that can guide education policy.

Key Contributions

1. A novel multi-level predictive model that systematically integrates four distinct SAEB data sources — student socioeconomic profiles, teacher professional data, consolidated school indicators, and principal management profiles — joined on school and class identifiers.
2. A rigorous comparison of four tree-based ensemble algorithms (Random Forest, XGBoost, LightGBM, CatBoost), in which Random Forest achieved 90.2% accuracy and an AUC of 0.9669 on the held-out test set.
3. A data-driven feature importance analysis identifying the school's average socioeconomic level as the single most dominant predictor, and demonstrating that systemic school-level factors outweigh isolated individual characteristics.
4. Actionable policy insights quantifying the systemic nature of academic achievement and pointing toward policies that reduce socioeconomic disparities between schools.

Main Findings

• Random Forest dominates: The Random Forest classifier reached 90.2% test accuracy and a test AUC of 0.9669. Its training accuracy was 0.9998 and training AUC 1.0000, a gap the authors attribute to the bagging methodology rather than harmful overfitting, given the test AUC.
• Gradient boosting models lagged: Run with default hyperparameters, CatBoost reached 64.17% test accuracy (AUC 0.6984), XGBoost 63.15% (AUC 0.6833), and LightGBM 62.48% (AUC 0.6743). Accuracies across the three ranged from 62.48% to 64.17%, with AUCs between 0.6743 and 0.6984.
• Balanced classification: For the "Above Average" class, precision was 0.91 and recall 0.89; for "Below Average", precision was 0.90 and recall 0.92. Weighted averages were 0.91 across precision, recall, and F1.
• Confusion matrix detail: Of 661,525 students with below-average performance, 608,603 were correctly identified (true negative rate 92.0%). Of 634,909 above-average students, 565,069 were correctly identified (true positive rate 89.0%).
• School socioeconomic composition is the top predictor: SHAP analysis identified the school's average socioeconomic level (NIVEL_SOCIO_ECONOMICO) as the most powerful feature.
• Compositional, not structural: The authors stress that this indicator is built exclusively from student questionnaire responses about parental education and household assets, aggregated to school level and grouped into Levels I through VIII. It does not capture physical infrastructure such as computers, libraries, or laboratories.
• Other systemic predictors: The percentage of teachers with adequate training and the student participation rate in the SAEB test also emerged as influential systemic variables.
• "Unknown" responses were meaningful: The SAEB category "unknown" or "not informed" (coded as category F) was retained as a valid class rather than treated as missing, and SHAP identified its negative predictive effect, reflecting a distinct potentially disadvantaged subgroup.
• Feature selection results: Boruta confirmed 44 of 98 candidate features as statistically relevant; an expert curation process reduced these to a final set of 17 features spanning all four analytical levels.
• Dataset scale: The final analyzed dataset contained 6,482,168 valid student records — 4,636,226 (71.52%) from 9th grade and 1,845,942 (28.48%) from high school — spanning 70,151 unique schools, 73,595 unique principals, and 290,972 unique teachers.

Methodology in Plain English

The researchers started with Brazil's official SAEB 2023 microdata, the most recent cycle available when data was accessed in January 2025 and the first post-pandemic national assessment under a fully restored administration protocol. They used four separate files — student, teacher, school, and principal — and joined them using school and class identifiers to build one large hierarchical table.

Identifiers were then dropped to prevent data leakage, rows with genuinely missing values were removed, percentage columns stored as text were converted to numbers, categorical variables were one-hot encoded, and all features were standardized with StandardScaler fit only on the training data. A key detail: the "unknown" answer about parental education was kept as its own category rather than discarded as missing.

The target was made binary: each student's Portuguese and Mathematics proficiency scores were averaged into a composite score, and students were labeled Above Average or Below Average relative to the dataset mean. The data was split 80% for training (5,185,734 records) and 20% for testing (1,296,434 records), stratified to preserve class ratios with a fixed random seed of 42.

Features were then narrowed using the Boruta algorithm, which compares each feature's importance against randomized "shadow features." Of 98 candidates, 44 were confirmed. From those, two researchers with Brazilian education policy expertise and two basic education practitioners selected a final set of 17 variables, prioritizing theoretical relevance, interpretability for policymakers, avoidance of redundancy, and representation across all four levels.

Finally, four ensemble classifiers were trained and evaluated on accuracy, precision, recall, F1, confusion matrix, and AUC-ROC. To explain the best model, the authors applied SHAP's TreeExplainer variant, which computes exact Shapley values from the internal tree structure and offers both computational efficiency and mathematical exactness at this data scale.

Why This Matters

Impact on research: The paper addresses a gap the authors identify in prior work — most Brazilian EDM studies focus on dropout in higher education at single institutions, while this study applies a systemic, multi-level approach to a recent large-scale national dataset for basic education. It also demonstrates a template for combining prediction with interpretability rather than treating accuracy as the endpoint.

Real-world applications:

• Targeted school investment: Identifying which schools have the lowest socioeconomic composition could help direct resources where disparities are greatest.
• Teacher training policy: The prominence of the percentage of teachers with adequate training as a predictor gives policymakers a concrete, school-level lever.
• Assessment participation campaigns: The influence of SAEB participation rates suggests value in efforts that raise student engagement with the assessment itself.
• Interpreting "unknown" responses: Treating unanswered parental education questions as a signal of disadvantage rather than noise could improve how schools identify students needing support.

Industry relevance: The workflow — integrating heterogeneous data sources, using Boruta for feature selection, comparing ensemble models, and applying SHAP for explanation — is directly transferable to domains such as finance, insurance, and healthcare, where accurate prediction must be paired with auditable reasoning.

The authors explicitly caution that the model identifies statistical associations, not causal relationships, and should be used as a formative tool for promoting equity rather than as a deterministic judgment about students.

Future Directions

• Disaggregating compositional from structural effects: The authors note that school socioeconomic composition and school physical resources are conceptually distinct though empirically correlated, and that future research separating them would yield additional policy insights.
• Hyperparameter tuning of the gradient boosting models: XGBoost, LightGBM, and CatBoost were run with default hyperparameters, so their substantially lower performance relative to Random Forest leaves open whether tuning would close the gap.
• Investigating the "unknown" response subgroup: Since this category showed a distinct negative predictive effect, further study of why students lack information about parental education could reveal additional equity dimensions.
• Broadening beyond the 17 curated features: The expert curation step deliberately excluded individual teacher and principal characteristics with low or redundant importance, so their conditional contribution under different modeling choices remains an open question.

Target Audience

This paper is most useful to education policy researchers and analysts working with national assessment data, educational data mining practitioners interested in multi-level modeling and interpretability, and Brazilian education administrators at the school or system level who want evidence about which systemic levers matter. Machine learning engineers new to applied social science data will also find the pipeline — from multi-source integration through Boruta selection to SHAP explanation — a readable end-to-end example, though the truncated feature importance table means the full ranked list of predictors is not available in the excerpt reviewed here.

Authors’ abstract

Identifying the factors that influence student performance in basic education is a central challenge for formulating effective public policies in Brazil. This study introduces a multi-level machine learning approach to classify the proficiency of 9th-grade and high school students using microdata from the System of Assessment of Basic Education (SAEB). Our model uniquely integrates four data sources: student socioeconomic characteristics, teacher professional profiles, school indicators, and principal management profiles. A comparative analysis of four ensemble algorithms confirmed the superiority of a Random Forest model, which achieved 90.2% accuracy and an Area Under the Curve (AUC) of 96.7%. To move beyond prediction, we applied Explainable AI (XAI) using SHAP, which revealed that the school's average socioeconomic level is the most dominant predictor, demonstrating that systemic factors have a greater impact than individual characteristics in isolation. The primary conclusion is that academic performance is a systemic phenomenon deeply tied to the school's ecosystem. This study provides a data-driven, interpretable tool to inform policies aimed at promoting educational equity by addressing disparities between schools.