EVOKE: Emotion Vocabulary Of Korean and English

AI summary

Overview

Research area: Natural Language Processing and cross-linguistic lexical semantics, specifically emotion-word resources (a dataset/annotation paper rather than a modeling paper).

Technical level: Intermediate. No model architecture or training is involved, but the paper assumes familiarity with annotation schemes, inter-annotator agreement (Cohen's kappa), and lexical semantics concepts such as polysemy, lexical gaps, and conceptual metaphor.

Scope: The paper introduces EVOKE, a Korean-English parallel dataset of 1,426 Korean and 1,397 English words with many-to-many translation mappings and a 14-criterion, theory-agnostic annotation scheme applied to adjectives and verbs in both languages.

What This Paper Is About

Emotion words are hard to pin down: some plainly name feelings (sadness, anger), while others describe bodily sensations, evaluations, or behaviors that may or may not count as emotions. Existing datasets each pick one theoretical definition and cover only a subset of words, and cross-linguistic resources are usually limited to fewer than 30 supposedly universal terms. This paper builds EVOKE, a Korean-English dataset that instead covers the widest possible pool of candidate emotion words and annotates each one against many different criteria, so researchers can apply whichever definition of "emotion word" they prefer, plus explicit many-to-many translations and polysemy/metaphor annotations.

Key Contributions

1. A comprehensive parallel lexicon. EVOKE contains 1,426 Korean words (591 nouns, 606 adjectives, 229 verbs) and 1,397 English words (508 nouns, 671 adjectives, 218 verbs), with many-to-many translational mappings between the two languages established by bilingual experts using bilingual and monolingual dictionaries.

2. A theory-agnostic 14-criterion annotation scheme. Candidate words were annotated across four parts: acceptability judgments in four sentence frames (acpt 1–4), semantic experiencer judgments (exp 5–8), exclusion criteria (excl 9–11), and multiple meanings (poly 12–14). All judgments were binary (acceptable/unacceptable, yes/no) with an "unsure" option, and criteria were translated and adapted for Korean syntax.

3. Systematic identification of lexical gaps. Words lacking a single-word translational equivalent in the other language were flagged: 52 Korean words (3.65%) and 59 English words (4.22%) had no translation equivalent, distinguishing true lexical gaps from morphosyntactic differences.

4. Annotation of polysemy and emotion-related metaphor. For each additional meaning identified (poly 12), annotators judged whether the senses belong to different domains (poly 13, distinctiveness) and whether they are systematically related (poly 14, relatedness), following the Metaphor Identification Protocol.

The dataset is publicly available at https://github.com/yoonwonj/EVOKE.

Main Findings

• Translational coverage was high and mostly many-to-many. 1,374 Korean words (96.35%) and 1,338 English words (95.80%) had at least one translation in the other language. Korean words averaged 1.61 English equivalents (SD = 0.93); English words averaged 1.65 Korean equivalents (SD = 1.15). 472 English words (33.79%) and 542 Korean words (38.01%) had two or more equivalents. The Korean adjective 우울하다 (uulhata) had six English equivalents: depressed, dismal, gloomy, low, down, and blue.

• Lexical gaps differed by part of speech. Korean gaps: 17 nouns (2.88%), 19 adjectives (3.14%), 16 verbs (6.99%). English gaps: 13 nouns (2.56%), 41 adjectives (6.11%), 5 verbs (2.29%). Examples include Korean 답답하다 (taptaphata), which has no single English equivalent and no conventionalized phrasal one, and English moody, which does have a multi-word Korean equivalent (기분 변화가 심한, kipun byŏnhwaka simhan). The authors state that systematically separating these two types of gap is beyond the scope of the paper.

• Korean annotations had higher inter-annotator agreement. Mean Cohen's κ was 0.74 for Korean and 0.60 for English. Exclusion criteria scored highest (Korean κ = 0.89, English κ = 0.82). Causality criteria scored lowest (exp 7–8: Korean mean κ = 0.52, English mean κ = 0.45), with the "causing" criterion exp 8 particularly difficult (Korean κ = 0.46, English κ = 0.45). Acceptability judgments: Korean mean κ = 0.73, English mean κ = 0.58. Subjective experience/evaluation (exp 5–6): Korean mean κ = 0.76, English mean κ = 0.47.

• "I am X" was more acceptable than "I feel X" in both languages. For adjectives, acpt 4 ("I am") was accepted for 91.83% of Korean and 86.85% of English words, versus acpt 1 ("I feel") at 82.13% Korean and 67.23% English, and acpt 2 ("They feel") at 84.28% Korean and 71.91% English. Most Korean adjectives were unacceptable in "It feels X" (83.36%), but English adjectives were not (55.14% unacceptable).

• Causality judgments were split, exclusions were consistent. exp 7 ("caused") was yes for 69.18% of Korean and 51.56% of English adjectives; exp 8 ("causing") was yes for 46.69% Korean and 51.82% English. Exclusion criteria showed roughly a 90% rejection rate in both languages.

• English adjectives were annotated as having multiple meanings more often. 12.68% of English adjectives versus 7.48% of Korean adjectives. Relatedness judgments differed (poly 14: Korean 87.37%, English 55.00%) while distinctiveness judgments converged (poly 13: Korean 89.47%, English 92.22%). Having an additional meaning correlated positively with bodily meanings, more strongly in Korean (r = 0.46) than English (r = 0.26), which the authors describe as consistent with metaphorical extension from physical to emotional domains.

• A conservative criterion yields 425 Korean and 317 English single-word emotion adjectives. Applying prior studies' inclusion rules — acceptable in "I feel", "They feel", and "I am" but not "It feels" (yes for acpt 1, 2, 4; no for acpt 3), and if acceptable in "It feels" then expressing evaluation rather than an inanimate subject's experience (no for exp 5, yes for exp 6), and not denoting pure bodily sensation, behavioral expression, or non-emotional epistemic state (no for excl 9–11) — produced these counts. Excluded examples include mean (못되다, mottwaeta in Korean), which feels unnatural in "I feel X", and clueless (어리둥절하다, ŏritungchŏlhata in Korean), which denotes a non-emotional epistemic state.

• Note on a reported discrepancy. The abstract states that 924 English words were annotated, while Section 5.2 reports 923 English words (678 adjectives, 245 verbs) annotated. The paper attributes discrepancies between annotated counts and translation-mapping counts to post-hoc corrections to part-of-speech coding and translation mappings.

Methodology in Plain English

The researchers started by gathering candidate words from prior English emotion-word studies and datasets (Baron-Cohen et al.; Morgan and Heise; Storm and Storm) and prior Korean ones (Jeon et al.; Park and Min; Rhee and Ko). Two Korean-English bilingual speakers then decided, word by word, whether each item had a translation equivalent in the other language. They did this manually rather than via machine translation, consulting one Korean-to-English and one English-to-Korean bilingual dictionary plus Korean and English monolingual dictionaries, comparing definitions to confirm that a match really captured the same meaning. A word counted as having an equivalent only if the other language had a single word form for it; if only multi-word phrases or idioms existed, or if back-translation was inconsistent, the item was treated as a lexical gap (with morphosyntactic differences excluded from this definition).

For annotation, the team recruited three native English speakers and three native Korean speakers with backgrounds in cognitive science or linguistics. They trained for one week and annotated for ten weeks, meeting weekly to discuss edge cases, which led to guideline revisions. Only adjectives and verbs were annotated (nouns were excluded because the standard sentence frames are built around predicative uses): adjectives received all 14 criteria, verbs received 9, excluding acpt 1, acpt 2, acpt 3, exp 5, and exp 6, with acpt 4 modified for verbs. Every judgment was binary with an "unsure" option; the authors chose binary over graded ratings because some questions are intrinsically categorical and to avoid idiosyncratic scale use. Unsure items were revisited in weekly meetings, but consensus was not forced. Each annotator saw a unique 30% of the words, and an additional 10% was assigned to all three annotators to measure agreement, with majority vote used to produce the combined analysis set. Korean annotations used translated sentence frames, with three different Korean subject forms allowed for the inanimate "it" frame and natural inflectional endings permitted.

Why This Matters

Impact on research. Prior resources either covered only small, supposedly universal sets of emotion terms or applied a single theoretical definition, making datasets hard to compare or combine. EVOKE separates the raw, comprehensive word pool from the selection criteria, so a researcher can reconstruct a "conservative" emotion-word list, a broad one, or a bespoke one, without re-annotating. Its bilingual annotations also let researchers ask whether the same semantic properties (subjective experience, causality, metaphor structure) behave the same way in Korean and English.

Real-world applications named or implied by the paper:

• Lexicon-based sentiment analysis and emotion detection, which rely on dictionaries of emotion words.
• Gold-standard translation data for NLP, since the many-to-many mappings were validated by bilingual speakers and can serve as a reliable benchmark for machine learning approaches and multilingual models.
• Computational analysis of lexical gaps, using emotion-word embeddings to see how multilingual models represent language-specific words such as 정 (chŏng) and 한 (han).
• Behavioral and cross-cultural psychology experiments, testing how conceptual gaps affect communication and interpretation of emotions, and whether shared dimensions like valence and arousal organize the emotion lexicon across languages.

Industry relevance. Developers building sentiment or emotion classifiers for Korean and English can use the annotated criteria to filter noise words (bodily sensations, behaviors, epistemic states) out of their lexicons; teams working on machine translation or cross-lingual embeddings get human-verified many-to-many word alignments; and product teams doing cross-market emotion analytics get explicit documentation of which English and Korean words do not line up one-to-one.

Future Directions

• Extend the approach to more languages. The paper explicitly notes that although the dataset covers two languages, many others remain to be investigated for their emotion words.
• Broaden multi-annotator coverage. Most items were annotated by a single trained annotator, with only about 10% of adjectives and verbs adjudicated across three annotators. Future work could increase the proportion of multiply-annotated items.
• Distinguish lexical gaps from conceptual gaps empirically. The identified gaps include both cases with no phrasal equivalent and cases with conventionalized multi-word descriptions (such as moody in Korean), and the authors state that separating these systematically is beyond the scope of this paper.
• Test the universality of emotion-related metaphors. The polysemy and relatedness annotations (poly 12–14) open the question of which metaphorical extensions are shared across cultures and which are language-specific.

Target Audience

Researchers in emotion science, psycholinguistics, computational linguistics, and NLP who need a Korean or English emotion-word resource; developers building sentiment analysis or emotion detection systems for either language; and cross-cultural researchers studying whether emotion concepts, lexical gaps, and metaphors are shared or language-specific. The paper is most useful to readers who want to select subsets of emotion words under an explicit theoretical criterion rather than consume a fixed word list.

Authors’ abstract

This paper introduces EVOKE (Emotion Vocabulary of Korean and English), a Korean-English parallel dataset of emotion words. The dataset offers comprehensive coverage of emotion words in each language, in addition to many-to-many translations between words in the two languages and identification of language-specific emotion words. The dataset contains 1,426 Korean words and 1,397 English words, and we systematically annotate 819 Korean and 924 English adjectives and verbs. We also annotate multiple meanings of each word and their relationships, identifying polysemous emotion words and emotion-related metaphors. The dataset is, to our knowledge, the most systematic and theory-agnostic dataset of emotion words in both Korean and English to date. It can serve as a practical tool for emotion science, psycholinguistics, computational linguistics, and natural language processing, allowing researchers to adopt different views on the resource reflecting their needs and theoretical perspectives. The dataset is publicly available at https://github.com/yoonwonj/EVOKE.