Sentiment, Emotion, Stance, and Aspect Analysis

A product review can hold several evaluations at once, and a document-level label compresses them badly. The constructs underneath are hard enough that trained people disagree about them. Three judges classified the same 270 tweets as sarcastic, positive or negative. They agreed unanimously on 135 of them. Credit only those unanimous judgements and accuracy over the whole test set falls to 43.33%.

Using “sentiment” as a catch-all creates invalid interpretations. The distance between two of these constructs has been measured rather than asserted. On the SemEval-2016 stance data, Mohammad and colleagues built an oracle. It is handed the gold sentiment label of every tweet, and it may then pick the best sentiment-to-stance mapping for each target. Perfect sentiment, converted as favourably as it can be converted, reaches F-macroT 53.1 and F-microT 57.2. The shared task's winning stance system reached F-macroT 56.0 and F-microT 67.8. A system trained to predict stance beats flawless sentiment labels. The authors say why: “This shows that even though sentiment can play a key role in detecting stance, sentiment alone is not sufficient.”

So the four constructs below are not four names for one judgement. Sentiment evaluates a target as positive, negative, neutral or graded, and can be mixed. Emotion identifies expressed or described affect; there the holder matters, and expression differs from experience. Stance determines support, opposition or uncertainty toward a proposition or entity that may never appear in the text. Subjectivity separates opinion and evaluation from more factual reporting, clause by clause, with quotation complicating who owns the view.

A single polarity label discards most of this structure. It discards who expresses or is attributed the evaluation, which entity or feature is evaluated, and which words support the label. It discards the direction and the degree. It discards the context too: negation, modality, time, comparison, irony and quoted source.

“Unpredictable” can be negative for a vehicle, positive for a thriller, and merely descriptive in a scientific report. “Positive” in medicine can indicate a finding rather than approval.

Domain adaptation needs labeled examples, target-aware evaluation, and phrase-level context. Blindly importing a general lexicon can produce confident category errors.

SemEval-2014 Task 4 put a number on the gap by holding everything else fixed. It released 7,686 hand-annotated review sentences in two domains: 3,841 restaurant sentences (3,041 training, 800 test) and 3,845 laptop sentences (3,045 and 800). One scheme covered both, with four aspect-term polarity values — positive, negative, neutral, and conflict for terms evaluated both ways at once. Same extraction subtask, same competitors, same year. Best F1 was 84.01% on restaurants and 74.55% on laptops. The task report states it plainly: “Overall, the systems achieved significantly higher scores (+10%) in the restaurants domain, compared to laptops.” Nothing changed but the things being reviewed.

The scheme should preserve disagreement rather than force annotators to infer unsupported intent. Disagreement here is not a rounding error. Three trained annotators worked over the same 210 sentences of world-press text under the MPQA scheme, and in 2005 Wiebe and colleagues reported what they found: “In the 210 sentences in the annotation study, the annotators A, M, and S respectively marked 311, 352 and 249 expressive subjective elements.” One text, three trained people, a 41% spread in how many opinion spans it contains. Average pairwise agreement was 0.72. It rose to 0.80 for medium-or-higher intensity spans and 0.88 for high-or-extreme ones. Agreement on the explicit private-state and speech-event anchors — the ones with a word you can point at — was higher, at 0.82. The annotators rated 89% of editorials as difficult and 73% of objective-topic articles as easy.

Read those numbers as a design brief for the five steps. Define the construct and the unit: document, sentence, clause, holder–target pair, or aspect. Mark holder, target and evidence, separating quoted, author and user-attributed opinions. That is the part that reached 0.82 rather than 0.72, because it is anchored to visible words. Specify context rules for negation, comparison, modality, mixed sentiment, irony and uncertainty. Include cannot-determine labels for ambiguity and insufficient context. A scheme that forces one answer on the sentences in the editorial 89% does not produce agreement; it hides the 311-versus-249 disagreement inside a single number. Then audit group and domain behavior across dialect, language, topic, source and consequence slices.

Document accuracy can hide wrong targets, holders or evidence. Report aspect, holder, span, orientation, intensity and attribution errors where the schema includes them.

SemEval-2016 Task 6 shows what one headline score conceals. It annotated 4,870 English tweets for stance towards six US targets — 2,914 training and 1,249 test instances across five targets in Task A, plus 707 Donald Trump tweets in Task B — and 19 teams entered. Now split the test set by whether the opinion expressed in the tweet is aimed at the target of stance or at some other entity. One system becomes two. The winning system scores F-macroT 59.7 on the tweets whose opinion is directed at the target and 35.4 on those directed at another entity (F-microT 72.5 against 44.5). The organisers' own SVM scores 62.5 against 37.9 (75.0 against 43.0). Their abstract records it: “However, systems found it markedly more difficult to infer stance towards the target of interest from tweets that express opinion towards another entity.” A single average conceals the split between the cases where the target is on the surface and the cases where it has to be reasoned about.

The same effect appears when one paper reports two accuracies. Socher and colleagues report that their model “pushes the state of the art in single sentence positive/negative classification from 80% up to 85.4%”. Separately, “The accuracy of predicting fine-grained sentiment labels for all phrases reaches 80.7%, an improvement of 9.7% over bag of features baselines”. One number is a document-level binary call. The other is a label on every phrase in a parse tree. Report whichever one the product actually consumes, and say which it is.

For monitoring or aggregation, examine prevalence, calibration, sampling bias, and who chooses to submit feedback. A dashboard summarizes recorded language, not the complete population’s feelings. In the US that has been more than methodological advice since 21 October 2024, when the FTC's rule on consumer reviews and testimonials, 16 CFR Part 465, took effect. Under §465.7(b) it is an unfair or deceptive practice “For a business to materially misrepresent, expressly or by implication, that the consumer reviews of one or more of the products or services it sells displayed in a portion of its website or platform dedicated in whole or in part to receiving and displaying consumer reviews represent most or all the reviews submitted to the website or platform when reviews are being suppressed (i.e., not displayable) based upon their ratings or their negative sentiment.” The rule leaves room only for sentiment-blind withholding criteria: confidential information, abusive or obscene content, suspected fakes, off-topic reviews. Note what the rule has isolated. The offence is not a wrong polarity label. It is a filter upstream of every polarity label, selecting on the very quantity the dashboard then averages.

Take a claim such as “Customers hate the new interface.” Specify the sampled source, the time, the aspect, the holder, the label definition, the uncertainty and the missing population. For that last one, name the component that did the excluding — as langid.py did at 13.2% against 7.6%, and as the filter on fashionnova.com did for four years.

Then design a model output that returns target, evidence, orientation and cannot-determine status. Say what the product must not conclude. Not a stance: perfect sentiment labels reached only F-macroT 53.1 against the stance system's 56.0. Not a diagnosis: Recital 44 of Regulation (EU) 2024/1689 records “serious concerns about the scientific basis of AI systems aiming to identify or infer emotions”. And not a claim about a population, because the denominator was assembled by components that no sentiment metric evaluates.