Exa-PSD: a new Persian sentiment analysis dataset on Twitter

AI summary

Overview

Research area: Natural Language Processing — Persian-language sentiment analysis and dataset construction for social media text.

Technical level: Intermediate. Readers should be comfortable with concepts such as annotated datasets, classification labels, and pre-trained transformer language models.

Scope in one sentence: The paper introduces and evaluates Exa-PSD, a 12,000-tweet Persian Twitter dataset annotated for three-class sentiment and benchmarked with two pre-trained language models.

What This Paper Is About

Sentiment analysis systems need training data that matches the way people actually write, but most existing Persian sentiment datasets are built around narrow topics such as products, foods, or hotels. That leaves a gap for the messy, informal language of social media, where Persian users employ irony and colloquial phrasing. The paper's goal is to fill that gap by constructing a Persian sentiment dataset drawn directly from Twitter and testing whether it is usable for training a sentiment system.

Key Contributions

1. Introduces a new Persian Twitter sentiment dataset. The authors present the Exa sentiment analysis Persian dataset (Exa-PSD), collected from Persian-language tweets.
2. Provides multi-annotator, native-speaker labeling. The dataset contains 12,000 tweets, each annotated by 5 native Persian taggers.
3. Uses a three-class label scheme. Tweets are labeled positive, neutral, or negative.
4. Reports dataset characteristics and a baseline evaluation. The paper documents statistics of the dataset and evaluates it using the pre-trained ParsBERT and RoBERTa models as base models.

Main Findings

• Dataset scale and composition: The released dataset consists of 12,000 Persian tweets.
• Annotation setup: Labeling was performed by 5 native Persian taggers across three sentiment classes (positive, neutral, negative).
• Baseline performance: Evaluation using pre-trained ParsBERT and RoBERTa reached a 79.87 Macro F-score.
• Authors' interpretation: On the basis of that score, the authors argue the model and data are adequately valuable for a sentiment analysis system.
• Details not in the abstract: The abstract does not report inter-annotator agreement, class distribution, per-class scores, train/validation/test splits, or individual results for each of the two models.

Methodology in Plain English

The authors gathered Persian tweets to build a sentiment dataset, then had five native Persian speakers label each tweet as positive, neutral, or negative. They describe the dataset's characteristics and statistics in the paper. To check whether the dataset is useful, they fine-tuned two existing pre-trained Persian-capable language models, ParsBERT and RoBERTa, on it and measured performance using Macro F-score — a metric that averages performance across the three sentiment classes rather than letting one dominant class dominate the score. The reported result was 79.87 Macro F-score. The abstract does not describe the collection procedure, annotation guidelines, filtering, or any agreement analysis.

Why This Matters

Impact on research: Persian NLP has comparatively few resources compared to higher-resource languages, and the abstract states that available Persian datasets tend to be topic-specific rather than drawn from general social media. Exa-PSD offers a Twitter-based, multi-annotator resource for Persian, which gives researchers a common benchmark and a source of training data for a language where such data is scarce.

Real-world applications (as suggested by the paper's framing):

• Monitoring public opinion on Twitter around specific topics or events.
• Brand and product reputation tracking based on what Persian-speaking users post.
• Social listening and trend analysis for Persian-language audiences.
• Content moderation or community sentiment dashboards that need to handle informal Persian.

Industry relevance: Companies and organizations that operate in Persian-speaking markets depend on sentiment tooling that handles colloquialisms and irony. A labeled dataset plus a demonstrated baseline pipeline lowers the barrier to building such systems, and an openly described benchmark gives teams a way to compare their own models.

Future Directions

• Addressing informal language explicitly. The abstract highlights irony and colloquial phrases as challenges; follow-up work could examine how well models handle these specific phenomena, which the abstract's evaluation does not break out.
• Comparing model architectures more thoroughly. Only ParsBERT and RoBERTa are mentioned as base models, leaving room to test other Persian or multilingual pre-trained models against the same benchmark.
• Expanding and diversifying the dataset. Twelve thousand tweets from one platform and one period may not cover the full range of Persian online expression; extending the resource is a natural next step.
• Studying annotation quality. With five taggers per tweet, reporting and analyzing agreement, and how disagreement is resolved, would clarify how reliable the labels are — something the abstract does not cover.

Target Audience

Researchers and practitioners working on Persian or low-resource-language NLP, especially those building sentiment classifiers or needing labeled social media data. It is also relevant to graduate students looking for a documented dataset-and-baseline paper, and to industry teams evaluating whether Persian sentiment analysis is feasible with existing pre-trained models.

Authors’ abstract

Today, Social networks such as Twitter are the most widely used platforms for communication of people. Analyzing this data has useful information to recognize the opinion of people in tweets. Sentiment analysis plays a vital role in NLP, which identifies the opinion of the individuals about a specific topic. Natural language processing in Persian has many challenges despite the adventure of strong language models. The datasets available in Persian are generally in special topics such as products, foods, hotels, etc while users may use ironies, colloquial phrases in social media To overcome these challenges, there is a necessity for having a dataset of Persian sentiment analysis on Twitter. In this paper, we introduce the Exa sentiment analysis Persian dataset, which is collected from Persian tweets. This dataset contains 12,000 tweets, annotated by 5 native Persian taggers. The aforementioned data is labeled in 3 classes: positive, neutral and negative. We present the characteristics and statistics of this dataset and use the pre-trained Pars Bert and Roberta as the base model to evaluate this dataset. Our evaluation reached a 79.87 Macro F-score, which shows the model and data can be adequately valuable for a sentiment analysis system.