Fine-Tuned Thoughts: Leveraging Chain-of-Thought Reasoning for Industrial Asset Health Monitoring

AI summary

Overview

• Research area: Natural language processing, specifically knowledge distillation and Chain-of-Thought (CoT) reasoning applied to industrial asset health monitoring and Failure Modes and Effects Analysis (FMEA) in Industry 4.0.
• Technical level: Intermediate. The paper assumes familiarity with large language models, in-context learning, and parameter-efficient fine-tuning (LoRA/QLoRA), but the domain logic is explained in plain terms.
• Scope: The paper (arXiv:2510.18817v1 [cs.CL], 21 Oct 2025, by Shuxin Lin, Dhaval Patel, and Christodoulos Constantinides of IBM Research) builds a seed-free synthetic data pipeline that distills CoT reasoning from large teacher LLMs into 8B-parameter small language models to answer FMEA multiple-choice questions about sensors and failure modes.

What This Paper Is About

Small Language Models (SLMs, defined in the paper as models with 8B parameters or fewer) are attractive because they are cheap to run and can be hosted locally, but their limited parameter capacity makes complex reasoning in specialized domains such as industrial asset health difficult. The authors' goal is to transfer reasoning ability from large teacher LLMs to small student models so that the students can reason about which sensors detect which failure modes without exposing sensitive industrial data to a hosted model. They do this by generating synthetic multiple-choice question answering (MCQA) data with Chain-of-Thought rationales, without relying on any initial seed documents, then fine-tuning the small models on it.

Key Contributions

1. A novel distillation framework that semi-automatically transfers Chain-of-Thought reasoning on multiple-choice question answering tasks from LLMs to SLMs.
2. A Knowledge Graph-inspired method that generates synthetic instructions, including pseudo-labels, for the industrial domain completely without seed documents.
3. A thorough qualitative evaluation of in-context learning and fine-tuning using the framework's generated domain knowledge, reporting that student models achieve performance improvements ranging from 11% to 23% depending on the base model.
4. Release of the FailureSensorIQ codebase and the FailureSensorIQ dataset for testing an LLM's ability to reason about sensor and failure mode relations.

Main Findings

• Fine-tuning closes much of the gap to large models: Fine-tuning on the generated CoT data produces P_single-correct gains of 0.11, 0.23, and 0.16 across the three student models. Llama-3.1-8B-Instruct fine-tuned on the CoT-Standard dataset reaches P_single-correct = 0.5111, comparable to the Llama-3.1-405B-Instruct baseline at 0.5126.
• Baseline hierarchy: Among baselines, Llama-3.1-405B-Instruct has the highest P_single-correct (0.51) and Mistral-Large-Instruct is close at 0.50, though Mistral-Large-Instruct has a higher P_invalid (0.024). Ministral-8B-Instruct (0.264) and Granite-3.1-8B-Instruct (0.2411) perform worst, with much higher P_mul-correct values, meaning they often select multiple answers rather than one precise answer.
• No clear winner among CoT styles: CoT-Standard, CoT-Expert, and CoT-Inductive each help, with the authors arguing CoT-Expert has a slight edge because it rarely generates invalid responses and keeps higher P_single-correct and P_mul-correct.
• Direct prompting wins after distillation: Once student models learn CoT-style reasoning through distillation, plain direct prompting most often yields the highest P_single-correct, meaning the models do not need explicit CoT prompts at inference time.
• In-context learning helps but plateaus: On 500 sampled FailureSensorIQ questions, moving from zero-shot to 5 expert-curated examples raised Llama-3.1-70B-Instruct from 249 to 303 correct inferences, and Mistral-Large-Instruct reached 320 with 5 curated examples. Gains plateau once the number of generated examples exceeds 20, and larger models tend to need fewer examples while smaller models benefit from more.
• Generated data is factually comparable to the benchmark data: Using an extended FActScore with Llama-4-Maverick on 700 samples evenly distributed across 54 asset types, teacher-generated data scored 70.8% versus 69.8% for FailureSensorIQ. The responding rate was 89.6% for generated data versus 94.7% for FailureSensorIQ.
• Rationales matter, and wrong rationales are harmful: Ablation shows incorrect pseudo-labeling (mismatched answer-rationale pairs) causes drops of 13.3% to a severe 21.4%. Removing rationales drops Llama-3.1-8B by 5.2% but slightly improves Ministral-8B, so the benefit of CoT fine-tuning is model-dependent.
• Models are unstable under knowledge-invariant perturbations: Under PertEval perturbations, fine-tuned Llama-3.1-8B-Instruct shows an average drop of 0.19 per perturbation versus 0.14 for Ministral-8B-Instruct. The KIQP (question paraphrasing) perturbation hurts more than OIDS (option ID shifting), indicating reliance on memorized patterns rather than deep context understanding.
• QLoRA is the practical fine-tuning choice: Full fine-tuning frequently degrades performance relative to LoRA because of a high proportion of invalid or incomplete responses, while LoRA and QLoRA achieve similar accuracy; QLoRA adds a lower memory footprint, faster training, and more efficient inference.

Methodology in Plain English

The authors define three FMEA relations that matter for asset health: mountedOn (a sensor is mounted on an asset), experiencedBy (a failure mode is experienced by an asset), and detectedBy (a failure mode can be detected by a sensor). Omitting the subject or object of such a relational triplet turns the remaining element into a seed, and handcrafted templates turn seeds into natural-language questions. In total there are 23 distinct seed templates covering four question categories: asset-to-sensors, failure-mode-to-class, failure-mode-to-sensor, and sensor-to-failure-mode.

For each question, the teacher LLM ranks candidate answers from a universal option set according to "correctness criteria." The top K options (K = 5) each become a correct answer in a separate, slightly rephrased question, which adds diversity and reduces bias; the bottom 2K options become distractor candidates, and answer positions are randomized. Pseudo-ground-truth labels are assigned by majority voting among three LLMs (reported as Mixtral Large/Mistral Large, Llama-3.1 405b, and ChatGPT/GPT-4 in different sections of the paper), where a label is accepted only if two voters agree and both confidence scores exceed 90, using a "self-guess" prompt.

Chain-of-Thought rationales are then generated with three trigger styles: Standard ("Let me think step by step"), Inductive ("Let me think step by step as a reliability engineer"), and Expert ("Let's use step by step inductive reasoning, given the domain specific nature of the question"). A rationale is kept only if the LLM's answer matches the pseudo-label. Quality filtering removes generations exceeding LLM context length, drops the lowest 5% by minimum neighbor distance, removes "very easy" questions judged by an LLM-as-a-Judge on a 5-point difficulty scale, and removes "very poor" and "poor" outputs on a 5-point quality scale.

Students are then fine-tuned with QLoRA at 4-bit precision using FlashAttention2, bf16 and tf32 mixed precision, maximum sequence length 2048 tokens, packing enabled, 1 epoch, batch size 8, gradient accumulation over 2 steps, learning rate 2.0 × 10⁻⁴, constant learning rate scheduler, warmup ratio 0.1, on 2 NVIDIA A100 80GB GPUs.

Why This Matters

• Impact on research: The work shows that CoT distillation into small models is viable for a specialized industrial domain where labeled data is scarce, and it contributes an openly released dataset (FailureSensorIQ) and codebase for a domain that has lacked a qualitatively constructed validation dataset comparable to PubMedQA or chemical safety benchmarks.
• Real-world application — predictive maintenance: Knowledge of which sensor anomalies correspond to which failure modes (bearing wear, gear defect, unbalance, shaft misalignment, overheating) supports proactive maintenance before breakdowns.
• Real-world application — on-premises deployment: Because SLMs can be fine-tuned, hosted, and operated locally on computing machines, sensitive user data and domain information need not be exposed or leaked to a hosted LLM.
• Real-world application — resource-constrained monitoring: Reduced computational requirements allow faster inference and deployment on edge or resource-limited devices in manufacturing settings.
• Real-world application — maintenance and quality control: The authors note opportunities in maintenance and monitoring, process optimization, and quality control across manufacturing.
• Industry relevance: The low cost of QLoRA fine-tuning for SLMs, reported as less than 1 hour per experiment and under 4GB adapter size for 8B models, makes domain adaptation practical and scalable for industrial adopters.

Future Directions

1. Develop perturbation-aware training or incorporate more diverse perturbation scenarios into the synthetic data generation process, given the significant performance drops observed under OIDS and KIQP.
2. Build more robust, domain-sensitive evaluation techniques for low-resource and high-precision scientific applications, since large-scale human validation is infeasible and automated scientific truthfulness checks remain limited in reliability and domain coverage.
3. Expand the framework beyond the three prototypical FMEA relations (mountedOn, experiencedBy, detectedBy) to cover the broader FMEA relational space, since the limited relational coverage may affect generalizability.
4. Reduce the risk of the student model inheriting subtle inaccuracies from the teacher, particularly for less-documented or highly specialized knowledge.

Target Audience

This paper is most useful for industrial AI and Industry 4.0 practitioners, reliability engineers working with FMEA and sensor-based condition monitoring, and NLP researchers interested in knowledge distillation, Chain-of-Thought reasoning, or synthetic data generation for low-resource specialized domains. It also suits engineers deciding between fine-tuning strategies such as Full FT, LoRA, and QLoRA for small language models deployed on-premises.

Authors’ abstract

Small Language Models (SLMs) are becoming increasingly popular in specialized fields, such as industrial applications, due to their efficiency, lower computational requirements, and ability to be fine-tuned for domain-specific tasks, enabling accurate and cost-effective solutions. However, performing complex reasoning using SLMs in specialized fields such as Industry 4.0 remains challenging. In this paper, we propose a knowledge distillation framework for industrial asset health, which transfers reasoning capabilities via Chain-of-Thought (CoT) distillation from Large Language Models (LLMs) to smaller, more efficient models (SLMs). We discuss the advantages and the process of distilling LLMs using multi-choice question answering (MCQA) prompts to enhance reasoning and refine decision-making. We also perform in-context learning to verify the quality of the generated knowledge and benchmark the performance of fine-tuned SLMs with generated knowledge against widely used LLMs. The results show that the fine-tuned SLMs with CoT reasoning outperform the base models by a significant margin, narrowing the gap to their LLM counterparts. Our code is open-sourced at: https://github.com/IBM/FailureSensorIQ.