Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback

AI summary

Overview

Research area: Human-Computer Interaction / User Experience (UX) research, applied natural language processing (NLP), and AI-assisted analytics in enterprise software.

Technical level: Intermediate. The paper is written for UX practitioners and product analysts rather than NLP specialists, but it assumes familiarity with classification metrics (precision, recall, F1), survey instruments (NPS, SUS, UX-Lite), and basic statistics.

Scope: A long-term industrial case study at SAP describing a production pipeline that classifies, summarizes, and interprets large volumes of open-text user feedback, plus an empirical investigation of whether comment sentiment can stand in for measured user satisfaction (it cannot, at least not for negative comments).

What This Paper Is About

Enterprise software companies collect thousands of free-text user comments through UX surveys, but analyzing that volume manually is slow and inconsistent. This paper describes techniques developed over a multi-year UX measurement program to automatically label comments with topic categories, generate readable summaries of them with generative AI, and correctly interpret the sentiment those comments express. The central warning is that negative comments are routinely — and wrongly — treated as a proxy for user dissatisfaction.

Key Contributions

1. A validated ten-label topic taxonomy and multi-label classifier for open-text UX comments, trained on 8,757 human-annotated comments drawn from 40 products on the SAP Business Technology Platform.

2. A comparison of embedding approaches showing that a Transformer-based SBERT model outperforms a fine-tuned fastText model by 24.58% in micro-averaged F1, using identical training data.

3. A refined generative AI prompting workflow for producing per-category comment summaries, including the discovery that anchoring summaries to predefined topic labels and requiring the model to cite supporting comment IDs dramatically reduces hallucination and improves validatability.

4. An empirical demonstration that comment sentiment and measured satisfaction diverge asymmetrically — positive sentiment reliably indicates high satisfaction, but negative sentiment does not indicate low satisfaction — supported by significance tests across two independent surveys.

Main Findings

• SBERT beats fastText for domain-specific comment embeddings. Switching from pre-processed fastText vectors to SBERT contextual embeddings improved the micro-averaged F1 score by 24.58%, despite using the same data volume. The authors attribute this to SBERT's context-awareness, which makes aggressive text preprocessing counterproductive.

• Extreme gradient boosting is the best-performing classifier among multi-label logistic regression, k-nearest neighbors, support vector machines, and random forest, evaluated on micro-averaged F1. Models are trained as independent binary classifiers per label, which accommodates the highly varied multi-label combinations in the data (169 unique combinations, 89 occurring only once).

• Model performance varies substantially by topic. Licensing achieved the strongest results (F1 = 0.82), followed by Performance (0.76), while the residual "Other" category performed worst (F1 = 0.49) because it aggregates semantically heterogeneous content. Usability, despite being the most frequent label (30.34% of comments), scored only F1 = 0.64.

• Naive GenAI summarization produced biased, unreliable output. Initial general-summary prompts over-weighted negative sentiment, varied in style between runs, ordered topics chaotically, presented contradictory statements without conveying sentiment distribution, and hallucinated claims. Manually validating every generated statement proved too labor-intensive to be worthwhile.

• Anchoring prompts to predefined categories fixed most problems. The best approach generates one summary per topic category above a minimum comment threshold, requires the model to output the supporting comment IDs and verbatim text, skips products with fewer than 20 comments, and maintains positive/negative balance in selected snippets. The "General Feedback" category turned out to be essential for preserving positive phrasing, because positive comments tend to be generic ("It's great") and fit poorly into other topical labels.

• Positive sentiment is a strong indicator of high satisfaction. In the tutorial survey, 85% of respondents with positive comments gave a tutorial quality score above 9. In the application survey, P(satisfied or very satisfied | positive comment) = 92%.

• Negative sentiment is a weak indicator of dissatisfaction. In the tutorial survey, P(Promoter | negative comment) = 47.24%, statistically indistinguishable from the probability of not being a promoter (p = 0.854). In the application survey, 22.15% of respondents with negative comments still reported being satisfied or very satisfied.

• The explanation is behavioral, not statistical. Commenters are self-selecting (only 12% and 7% of survey respondents left comments in the two surveys). Positive comments are short and generic (average 35–53 characters); negative comments are long and specific (average 153–202 characters), typically describing isolated issues, error codes, or navigation problems that users hope will be fixed. A single bad experience dents but does not destroy overall satisfaction.

Methodology in Plain English

The team began by reading many existing comments and interviewing the people who would use the results, then drafted and iteratively refined a codebook of ten topic labels plus definitions. Two human coders independently applied these labels to a sample of comments, which established the training data and let the team check inter-coder reliability.

Each comment was then converted into a numerical vector (an embedding) using two different approaches for comparison: an older word-level model (fastText) with heavy text cleanup, and a modern sentence-level Transformer model (SBERT). Those vectors became the input features for a machine learning classifier. Because a single comment can touch several topics, the team trained one binary classifier per topic label using extreme gradient boosting, tuning hyperparameters through stratified k-fold cross-validation. After deployment, two human coders continuously corrected the model's predictions, and the model was periodically retrained on the growing pool of corrected labels — a human-in-the-loop setup that now spans 8,757 labeled comments.

For summarization, the team iterated through several prompt designs before settling on one that feeds the model both the comments and their assigned topic labels, asks for a summary per category, and requires the model to list which comment IDs supported each statement. Validation became a fast skim rather than a full audit, because unsupported claims were self-evident from the citation list.

Finally, the team tested whether comment sentiment tracked satisfaction by pairing each commenter's sentiment label (assigned by Qualtrics, with manual correction in the second survey) with their numeric survey scores, then running chi-squared tests, computing conditional probabilities, and plotting cumulative frequency distributions.

Why This Matters

This paper sits at the intersection of two practical problems that most large software organizations now face: too much qualitative feedback to read, and too much trust placed in automated text analytics. Its main contribution to the research literature is the empirical refutation of a convenient assumption — that sentiment ratios approximate satisfaction — using two independent, sizeable survey datasets with proper significance testing. That negative result is more useful to practitioners than another positive classification benchmark would be.

Real-world applications:

• Product backlog triage. Filtering comments by predicted topic (especially "Error") lets product managers surface actionable bugs and usability failures in minutes rather than days, and track whether the share of complaints about a given topic declines after an investment.

• Executive and stakeholder reporting. Quarterly and annual reports can include GenAI-generated, category-organized summaries with supporting verbatim quotes, replacing hundreds of raw comments with a page or two of validated text.

• UX metric interpretation guidance. Teams that currently panic over a rise in negative comment volume can now check whether the corresponding satisfaction scores moved — and often they will not.

• Dashboard-driven continuous monitoring. The pipeline feeds an interactive dashboard available company-wide, letting any employee see both quantitative trends and representative user voices.

Industry relevance: The work is directly transferable to any organization running continuous UX measurement at scale — enterprise software, SaaS, consumer platforms — and is particularly relevant to teams already using survey tools with built-in sentiment classification, since it cautions against over-reading those built-in labels.

Future Directions

• Improving classification for heterogeneous categories. The "Other" label performed poorly (F1 = 0.49), suggesting that either the taxonomy needs finer granularity or that residual categories need a different handling strategy.

• Quantifying the cost-benefit of human-in-the-loop retraining. The paper describes continuous correction and batch retraining but does not report how much predictive performance each retraining cycle actually buys, or how much human effort it costs.

• Generalizing the sentiment–satisfaction asymmetry. The finding rests on two surveys within one company's product ecosystem and two specific metrics (tutorial quality/NPS and PSAT/UX-Lite). Replication across other industries, survey instruments, and languages remains open.

• Reducing reliance on manual validation of GenAI summaries. The current workflow still requires human skimming and occasional manual rewriting. Whether stricter prompting, retrieval-augmented generation, or automated claim verification can reduce that burden without sacrificing reliability is an open engineering question.

Target Audience

UX researchers and product analysts who run large-scale feedback programs; data scientists building text classification pipelines for domain-specific enterprise content; product managers who consume survey reports and need to interpret comment sentiment correctly; and NLP practitioners interested in a real-world account of where Transformer embeddings and generative summarization succeed and fail in production.

Authors’ abstract

In highly competitive software markets, user experience (UX) evaluation is crucial for ensuring software quality and fostering long-term product success. Such UX evaluations typically combine quantitative metrics from standardized questionnaires with qualitative feedback collected through open-ended questions. While open-ended feedback offers valuable insights for improvement and helps explain quantitative results, analyzing large volumes of user comments is challenging and time-consuming. In this paper, we present techniques developed during a long-term UX measurement project at a major software company to efficiently process and interpret extensive volumes of user comments. To provide a high-level overview of the collected comments, we employ a supervised machine learning approach that assigns meaningful, pre-defined topic labels to each comment. Additionally, we demonstrate how generative AI (GenAI) can be leveraged to create concise and informative summaries of user feedback, facilitating effective communication of findings to the organization and especially upper management. Finally, we investigate whether the sentiment expressed in user comments can serve as an indicator for overall product satisfaction. Our results show that sentiment analysis alone does not reliably reflect user satisfaction. Instead, product satisfaction needs to be assessed explicitly in surveys to measure the user's perception of the product.