Language-Independent Sentiment Labelling with Distant Supervision: A Case Study for English, Sepedi and Setswana

AI summary

Overview

Research area: Natural Language Processing — sentiment analysis and distant-supervision data labelling for low-resource languages.

Technical level: Beginner-Friendly to Intermediate. The method relies on frequency counting and emoji matching rather than model training, so no deep learning background is required to follow the approach.

Scope: The paper proposes and evaluates a language-independent sentiment labelling algorithm that uses sentiment-bearing emojis and emoji-derived word lists to pre-label English, Sepedi and Setswana tweets, reporting how much manual annotation effort can be avoided.

What This Paper Is About

Many African languages are low-resource because labelled sentiment data is scarce, and manual labelling is slow and expensive. This paper asks whether a single algorithm that needs no language-specific knowledge can automatically pre-label sentiment in tweets, so that human annotators only have to correct mistakes instead of labelling everything from scratch. It tests this idea on 7,000 English, 7,000 Sepedi and 7,000 Setswana tweets drawn from the SAfriSenti corpus.

Key Contributions

1. A three-step, language-independent sentiment labelling algorithm that first labels tweets containing sentiment-bearing emojis, then automatically builds class-specific word lists from those tweets, and finally labels the remaining tweets by word overlap with those lists.
2. A demonstration that the approach requires no machine learning training at all — it is based purely on frequencies, avoiding manual or high computational effort.
3. An empirical evaluation across three languages (English, Sepedi, Setswana) with equivalent dataset sizes of 7,000 tweets per language, showing accuracies between 63% and 69%.
4. A quantified estimate of annotation savings: on average only 34% of the automatically generated labels remain to be corrected, with human annotators needing to change between 31% and 37% of pre-labelled tweets.

Main Findings

• Overall accuracy: The algorithm labels English tweets with an accuracy of 66%, Sepedi tweets with 69%, and Setswana tweets with 63%, so on average only 34% of the automatically generated labels remain to be corrected.
• Per-step accuracies (Table 6): Emoji-based labelling (step 1) reached 68.7% for English, 69.5% for Sepedi and 66.1% for Setswana. Word-list labelling (step 3) reached 63.5% for English, 64.6% for Sepedi and 59.7% for Setswana. The combined step 1-to-3 figures were 66.2% for English, 68.7% for Sepedi and 62.7% for Setswana.
• Macro F1-scores (Table 7): Step 1 scored 66.8% (English), 68.2% (Sepedi) and 65.2% (Setswana); step 3 scored 62.2%, 63.4% and 58.7% respectively; combined step 1-to-3 scored 64.6%, 67.9% and 61.5%. F-scores were computed as the Macro F1 score, the average of each class's F1 score.
• Emojis outperform words: The emoji step produces better labelling quality than the word-list step, indicating that the automatically created word lists are weaker indicators than the emojis themselves.
• Coverage varies sharply by language (Table 5): Tweets labelled by emojis in step 1 were 4,210 (60.1%) for English, 5,871 (83.9%) for Sepedi and 3,249 (46.4%) for Setswana. The remaining tweets handled by word lists in step 3 were 2,790 (39.9%), 1,129 (16.1%) and 3,751 (53.6%) respectively.
• More emoji coverage helps the word lists: The authors attribute Sepedi's strongest performance to the fact that 84% of the 7,000 Sepedi tweets contain sentiment-bearing emojis, compared with 60% for English and 47% for Setswana, giving more text for generating word lists. The paper also notes that "with an accuracy of 70%" the challenge of using emojis as a better indicator remains; this figure appears in the discussion alongside the 69% quoted in the abstract and the 68.7% in Table 6.
• Reduction of manual effort: With 66% accuracy, annotators would still have to change 34% of labelled tweets for English; Sepedi at 68% accuracy would require 32% changes; Setswana at 63% accuracy would require 37% of labels to be changed.
• Emoji inventory: The experiments used 12 negative, 10 neutral and 12 positive emojis, matched in Unicode for platform independence. If a tweet contains emojis of different sentiments, it is labelled with the sentiment class having the most emojis; if there is no majority, the tweet is not labelled at this step.

Methodology in Plain English

The algorithm works in three stages. First, each tweet is checked against a curated list of emojis known to carry negative, neutral or positive sentiment — 12 negative, 10 neutral and 12 positive emojis in total. Emojis are compared in Unicode so the method is not tied to a particular platform. Tweets are assigned a sentiment when one class has an emoji majority. Second, all the words from the tweets that were labelled in the first stage are collected per sentiment class, and any word that also appears in one or both of the other class lists is removed, leaving lists of words that are distinctive to each sentiment. Third, the remaining tweets — those without sentiment-bearing emojis — are labelled according to which of these word lists they overlap with most.

The data comes from SAfriSenti, described as the largest sentiment dataset available for South African languages, with 64.3% monolingual tweets in English, Sepedi and Setswana and 36.6% code-switched tweets between these languages. The experiments used 7,000 monolingual tweets per language. The English subset contains 2,052 positive (29.3%), 3,448 negative (49.3%) and 1,500 neutral (21.4%) tweets. The Sepedi subset contains 3,500 positive (50.0%), 2,270 negative (32.4%) and 1,230 neutral (17.6%) tweets. The Setswana subset contains 3,230 positive (46.1%), 2,180 negative (31.1%) and 1,590 neutral (22.8%) tweets. Performance was measured as accuracy and Macro F1 against the gold labels in that corpus.

Why This Matters

The work targets a practical bottleneck in sentiment analysis for under-resourced languages: the absence of labelled data, which stems from manual annotation being time-consuming and expensive. By showing that a single frequency-based, training-free procedure can pre-label text across three languages with accuracies between 63% and 69%, it suggests a cheaper route to bootstrapping datasets for languages that currently lack them. Because the approach assumes only that people use emojis regardless of the language they post in, the authors argue it should transfer to other languages.

Real-world applications:

• AI for Social Good: monitoring opinions and emotions at scale in citizen-generated text, an area the paper explicitly names alongside sentiment analysis generally.
• AI in Education: analysing learner feedback and opinions, also named by the authors as an application area.
• Marketing and business intelligence: the paper notes sentiment analysis draws attention in business environments for extracting polarity from textual feedback and opinions.
• Medicine: cited among the domains where sentiment analysis is applied.

Industry relevance: For organisations building sentiment products in African markets, the result offers a way to create initial training data without a large annotation budget. The authors quantify the commercial value directly: instead of adding all labels from scratch, annotators only need to change between 31% and 37% of pre-labelled tweets. Since the method is frequency-based and requires no training, it is also inexpensive to run.

Future Directions

• Extend to more languages and cross-lingual features: The authors expect the approach to work for other languages and want to experiment with cross-lingual features, noting that adding features such as translated tweets or word lists translated from other languages could help but would make the algorithm no longer completely language-independent, since it would draw on information from outside the corpus.
• Handle mixed-sentiment tweets: Investigate whether it is helpful to label a tweet as neutral when it contains a comparable number of positive and negative emojis — the current algorithm assigns no label in that case.
• Combine with active learning: Build an iterative process where annotators manually classify tweets that are then used to re-train machine learning systems for classification.
• Build a multilingual NLP model: The authors aim to create a multilingual model and investigate synergy effects across languages in sentiment analysis, suggesting prior word-embedding work as a possible basis.

Target Audience

Researchers and practitioners working on sentiment analysis for low-resource and African languages, particularly those who need labelled data but lack annotation budgets. It is also relevant to NLP engineers building data-labelling pipelines, to annotation teams estimating how much review work a pre-labelling system saves, and to readers interested in language-independent methods that avoid model training. The paper is accessible without deep machine learning expertise, since the method is based on frequency counting and emoji matching rather than neural architectures.

Authors’ abstract

Sentiment analysis is a helpful task to automatically analyse opinions and emotions on various topics in areas such as AI for Social Good, AI in Education or marketing. While many of the sentiment analysis systems are developed for English, many African languages are classified as low-resource languages due to the lack of digital language resources like text labelled with corresponding sentiment classes. One reason for that is that manually labelling text data is time-consuming and expensive. Consequently, automatic and rapid processes are needed to reduce the manual effort as much as possible making the labelling process as efficient as possible. In this paper, we present and analyze an automatic language-independent sentiment labelling method that leverages information from sentiment-bearing emojis and words. Our experiments are conducted with tweets in the languages English, Sepedi and Setswana from SAfriSenti, a multilingual sentiment corpus for South African languages. We show that our sentiment labelling approach is able to label the English tweets with an accuracy of 66%, the Sepedi tweets with 69%, and the Setswana tweets with 63%, so that on average only 34% of the automatically generated labels remain to be corrected.