FicSim: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fiction

AI summary

Overview

Research area: Natural Language Processing / computational literary studies — specifically long-context embedding evaluation and semantic textual similarity (STS) for fiction. Technical level: Intermediate (assumes familiarity with embedding models, Spearman correlation, and STS benchmarks). Scope: The paper introduces and releases FicSim, a dataset of 90 long-form fanfiction stories with gold similarity scores along 12 axes, and uses it to benchmark a suite of embedding models.

What This Paper Is About

Existing embedding similarity datasets focus on coarse-grained similarity between very short texts, which makes them unsuitable for evaluating models on literary tasks where scholars care about specific semantic facets such as plot, tone, setting, or theme. Public-domain literature is also a poor test bed because it is heavily present in model pretraining data and in online analyses, raising contamination concerns. The authors build FicSim from recently written, author-consented fanfiction with author-assigned tags, then measure whether current embedding models can capture fine-grained literary similarity.

Key Contributions

1. A new evaluation dataset for long-form literary STS. FicSim contains 90 stories, 33,790 pairwise comparisons, and gold similarity scores along 12 axes, built from author-produced tags on Archive of Our Own (AO3).
2. A tag-derived, scholar-validated annotation framework. The 12 similarity categories were developed with reference to digital humanities scholarship and categorized by two annotators with expertise in fan studies, computational literary studies, and library and information science, then validated through multi-stage triplet annotation.
3. An ethical data-collection protocol centered on author agency. The authors obtained IRB-approved individual consent from every author, committed to not using FicSim to train models, and require the same commitment from anyone accessing the dataset.
4. A benchmark of 7 embedding models on multi-faceted literary STS, showing that models over-index on surface features (author and fandom) rather than fine-grained semantic categories, and that larger or more expensive models are not uniformly better.

Main Findings

• Gold similarity scores weigh surface features far less than models do. Table 4 shows that FicSim's own category scores correlate comparatively weakly with the fandom-specific, fandom, and author categories, while embedding models score highest on those same confounding categories.
• Models over-index on authorial style. The author category, computed solely from exact match of author IDs, has the highest Spearman correlation more frequently than any other category. For example, Linq-Embed reaches 40.91 on Author but only 0.14 on Theme; GTE-Qwen2 reaches 40.82 on Author versus 13.92 on Theme.
• 77% of the models in Table 3 score higher in the fandom-specific, fandom, or author category than in the overall (fandom-specific) category — meaning they capture author and fandom fingerprints while failing to capture other narrative elements.
• Sliding windows do not improve fandom-agnostic correlations. Mean Spearman correlation under naive truncation is 11.75 versus 11.69 for the sliding-window mean, and truncation beats sliding window exactly 50% of the time. Sliding windows do reduce correlations in the confounding categories (mean ρ of 28.62 versus 36.53).
• Category-specific prompting yields only slight improvement. When comparing a given model and context-handling method, category-specific embeddings outperform non-specific embeddings 55% of the time.
• No single model performs well across all types of similarity. Voyage-3-large, for instance, scores 50.95 on Overall (Fandom-Specific) and 25.08 on Overall (Fandom-Agnostic), but only 5.72 on Character States, 7.11 on Theme, and 9.86 on Relationship Dynamics.
• Bigger and more expensive is not uniformly better. The 149M-parameter GTE-ModernBERT reaches 50.08 on Fandom-Specific, 48.50 on Overall (Fandom-Specific), and 40.24 on Time with a sliding window — competitive with or above 7B models on several categories.
• An LLM-summarization pipeline did not solve the task. Using Claude-3.7-Sonnet to summarize stories and then embedding the summaries with Voyage produced weak fine-grained results, including -1.19 on Plot and 27.54 on Time.
• Annotation agreement was moderate to good. Of 158 triplets that neither of two annotators skipped, annotator agreement was 82% (Cohen's κ = 0.65); gold labels aligned 80% with the 129 triplets on which both annotators agreed.

Methodology in Plain English

The researchers needed long narratives that are publicly available for evaluation, unlikely to appear in model pretraining data, and richly annotated. They sourced English-language fanfiction from Archive of Our Own, which hosts over 15M works and blocked Common Crawl scraping in 2022 and restricted web crawling in December 2022. They considered only stories started and completed after that date, exceeding 10,000 words, and with detailed tagging — yielding stories ranging from 10,001 to 488,772 words across 46 fandoms.

Fanfiction authors already tag their stories extensively to help readers find content, so the team treated those tags as gold-standard annotations. They cleaned and standardized the tags (removing punctuation, standardizing capitalization, correcting spelling, mapping to canonical forms) and sorted them into 12 categories. For the nearly 50% of tags containing fandom-specific references, they created fandom-agnostic rewrites — for example, turning "Protective Cristina Yang" into "protective character" — so that non-fandom categories could be used to compare stories across fandoms. The cleaned tagset contains 9448 total tags (2133 unique).

To convert tags into similarity scores, each tag was embedded with Gemini Embedding, and the similarity between two stories in a category was computed as the average of pairwise tag cosine similarities. Gemini Embedding was chosen because it had the highest overall MTEB leaderboard score, was the highest-ranking model not from the same family as the models being evaluated, and has a 2048-token input limit that would not preclude it from the evaluation.

The gold scores were validated in two stages: one author annotated 330 story triplets (thirty 3-way comparisons in each of the 11 non-author categories, drawn from a randomly generated pool of 100 triplets per category and filtered for ambiguity), and the tag cleaning and embedding pipeline was adjusted to align with those judgments. Two other authors then annotated 220 triplets each.

For evaluation, the team embedded all story pairs with 7 models and computed Spearman's ρ between the model-induced similarity ranking and the gold ranking, reporting ρ out of 100 with significance at p < 0.05. The models were Linq-Embed-Mistral, GTE-Qwen2-7B-instruct, and SFR-Embedding-Mistral (the best-performing 32k-context open-weights models on the Hugging Face MTEB leaderboard, all 7B), plus the smaller GTE-ModernBERT-base (149M), m2-BERT-80M-32k-retrieval via the TogetherAI API, Voyage-3-large via the Voyage.ai API, and a Claude-3.7-Sonnet summarization pipeline embedded with Voyage. Because some stories exceed every model's context length, they compared naive truncation against a sliding window that chunks the text and pools embeddings, with windows overlapped by 2048 characters (roughly 500 words).

Why This Matters

Literary scholars increasingly want to use NLP to find semantically similar passages, stories, or tropes, but the field lacks benchmarks that test what those scholars actually care about. FicSim shows that high-performing general embedding models largely latch onto who wrote a text and which fandom it belongs to — information that is trivially available from metadata — rather than the narrative content. That gap matters both for how researchers interpret existing results and for how model developers target future improvements.

Real-world applications:

• Selecting appropriate embedding models for retrieval, clustering, and similarity analysis over long fiction collections in digital humanities and library contexts.
• Building recommendation and discovery systems over large fiction archives, where tag-based categories such as "Enemies to Lovers" or "Slow burn" have been adopted outside fan spaces by traditional publishers and readers (Goodreads has added "Enemies to Lovers" as a book genre).
• Auditing retrieval and search systems for long documents, since current embeddings may rank items by authorial or stylistic fingerprints rather than topical or thematic relevance.
• Establishing consent-respecting norms for using creative writing in AI research datasets and benchmarks.

Industry relevance: Embedding model providers and teams building long-context retrieval pipelines have a concrete measurement of where current models fail on long narrative inputs. The finding that a 149M-parameter model is competitive with 7B models on several categories is directly relevant to cost and deployment decisions, and the paper's consent and usage-restriction framework offers a template for companies that want to use creative works without circumventing creators' wishes.

Future Directions

• Multiple-embedding strategies. The authors explicitly identify computing similarities using multiple-embedding strategies as a likely route to improvement, and leave better embedding strategies for literary-domain text to future work.
• Better adaptions for long-form inputs. The paper notes that sliding windows and category-specific instructions had limited impact, and suggests clever system design or additional domain-specific training could improve performance within this generation of embedding models.
• Domain generalization. FicSim covers only English fanfiction; the authors acknowledge that models excelling on FicSim might still struggle on STS tasks for 19th century English literature, short-form satirical poetry from social media, or other specialized literary domains.
• Responsible replication. Because the authors' AO3 outreach account was temporarily suspended for spam after they followed instructions given by an AO3 support member, they state that an exact replication of their process would not be appropriate — leaving open the question of how to scale consent-based creative-text datasets.

Target Audience

Digital humanities and computational literary studies researchers choosing models for similarity tasks; NLP researchers working on long-context and embedding evaluation; embedding model developers; and fan studies, media studies, and library and information science scholars interested in ethical dataset construction and the use of fan-created works in research.

Authors’ abstract

As language models become capable of processing increasingly long and complex texts, there has been growing interest in their application within computational literary studies. However, evaluating the usefulness of these models for such tasks remains challenging due to the cost of fine-grained annotation for long-form texts and the data contamination concerns inherent in using public-domain literature. Current embedding similarity datasets are not suitable for evaluating literary-domain tasks because of a focus on coarse-grained similarity and primarily on very short text. We assemble and release FICSIM, a dataset of long-form, recently written fiction, including scores along 12 axes of similarity informed by author-produced metadata and validated by digital humanities scholars. We evaluate a suite of embedding models on this task, demonstrating a tendency across models to focus on surface-level features over semantic categories that would be useful for computational literary studies tasks. Throughout our data-collection process, we prioritize author agency and rely on continual, informed author consent.