CascadeMind at SemEval-2026 Task 4: A Hybrid Neuro-Symbolic Cascade for Narrative Similarity

AI summary

Overview

Research area: natural language processing, specifically computational narrative understanding, uncertainty estimation in large language models, and neuro-symbolic system design.

Technical level: Intermediate. The four-stage cascade and voting logic are easy to follow; the symbolic ensemble involves TF-IDF, sentence-transformer embeddings, sentiment-curve correlation, and differential-evolution weight fitting.

Scope: This paper describes a competition system that decides which of two candidate stories is more similar to an anchor story, by using agreement among eight LLM samples as a routing signal and reserving a theory-inspired symbolic ensemble for only the hardest ties.

What This Paper Is About

Comparing two stories requires judging shared theme, sequence of events, and outcome, which is harder than matching surface wording. The authors build CascadeMind for SemEval-2026 Task 4, where a system sees an anchor story and two candidates and must say which candidate is closer to the anchor along abstract theme, course of action, and outcomes. Rather than making one LLM call, they sample many votes, use how much those votes agree as a difficulty signal, and spend extra compute only where the votes disagree.

Key Contributions

1. A four-stage cascade in which LLM vote agreement determines the decision pathway, producing accuracy that is monotone in vote consensus (85% for supermajority cases, 67% for split votes, 61% for perfect ties).
2. A symbolic ensemble of five theory-inspired similarity signals (lexical, story-grammar, semantic embedding, tension curve, event chain) used only as a fallback on perfect ties.
3. Ablations isolating where the gain comes from, showing that confidence-aware routing accounts for nearly all of it while the symbolic fallback contributes negligibly because only 5% of cases reach it.
4. An official shared-task result of 72.75% on Track A test, placing 10th of 44 teams, plus post-hoc diagnostics on released test labels.

Main Findings

• Vote agreement predicts difficulty. Across self-consistency samples, supermajority cases (at least 7/8 votes) resolve at 85% accuracy, split votes at 67%, and perfect ties at 61% — a monotone gradient that holds across the development set.
• Official standing. CascadeMind reached 72.75% on the Track A test set, placing 10th of 44 teams. The task received 71 submissions from 46 teams across both tracks.
• Cascade beats simpler baselines on development. On the development split, single vote scored 68.0% (1.0 calls/case), self-consistency at k=8 scored 76.5% (1.0 calls/case), majority over three calls scored 78.0% (3.0 calls/case), and CascadeMind scored 81.0% at 1.78 average calls/case. Adding the symbolic tiebreaker left the 81.0% figure unchanged.
• The symbolic fallback barely matters end-to-end. On a separate perfect-tie diagnostic set (n=18), the symbolic tiebreaker achieved 61.1% (11/18). Applied to all cases regardless of neural confidence, the symbolic ensemble dropped to 53% on the cascade diagnostic subset.
• Symbolless performance is above chance but below the cascade. As a standalone classifier with fixed paper weights and no LLM calls, the symbolic module scored 57.0% on development (114/200) and 60.5% on test (242/400).
• Class asymmetry on test. For A-closer cases, precision/recall/F1 were 76.9%/68.8%/72.6%; for B-closer cases, 69.6%/77.6%/73.4%. Macro-F1 was 73.0% and balanced accuracy 73.2%. The model predicted B 53.5% of the time versus A 46.5%, while released labels were 52.0% A and 48.0% B. The dominant error was predicting B when A was correct (65 cases).
• A substantial development-to-test gap. Accuracy fell from 81.0% on the cascade diagnostic subset (n=100) to 72.75% (291/400) in official evaluation. Post-hoc Track A diagnostics scored 73.0% (292/400), a one-prediction difference from the submitted file.
• Synthetic-data overfitting in the symbolic weights. The ensemble reached 99.5% on a held-out synthetic validation split (n=200) but only 61.1% on the perfect-tie diagnostic set. Weights concentrated on lexical (49%) and story-grammar (40%) signals over semantic embeddings (8%), with tension curve at 2% and event chain at 1%.

Methodology in Plain English

The system asks a model a direct comparative question — which candidate, A or B, is more similar to the anchor on theme, course of action, and outcomes — and expects a JSON decision field. No chain-of-thought rationale is requested or used.

It then runs four stages:

1. Sample eight votes. Gemini 2.5 Flash is called with candidateCount=8 at temperature 1.0, producing eight independent A/B votes.
2. Check confidence. If at least 7 of 8 votes agree, that answer is returned immediately.
3. Escalate splits. On 4-4, 5-3, or 6-2 splits, three more candidateCount=8 calls are issued for 32 votes total and a majority is taken.
4. Fall back only on perfect ties. A 16-16 tie invokes a symbolic ensemble of five weighted similarity signals: TF-IDF cosine lexical overlap (weight 0.49), phase-aligned story-grammar similarity across five narrative phases (setting 0-20%, conflict 20-40%, rising action 40-60%, climax 60-80%, resolution 80-100%) using sentence-transformer embeddings (0.40), whole-story semantic embedding cosine with all-MiniLM-L6-v2 (0.08), Pearson correlation of per-sentence tension curves derived from TextBlob sentiment intensity and subjectivity interpolated to 10 points (0.02), and normalized longest common subsequence over action-verb sequences filtered against a 47-word list (0.01).

The five weights were fitted with differential evolution on an organizers' 1,900-triplet LLM-generated synthetic split, used only for calibration. The authors report development and test results on the task's human-labeled splits: a 200-triplet Track A development split and a 400-triplet test split, drawn from Wikipedia plot synopses in the English portion of Tell-Me-Again, typically four to eight sentences long. Routing shares measured on a 100-case diagnostic subset were 74% supermajority, 21% escalated, and 5% symbolic, giving expected API usage of 1.78 calls per case.

Why This Matters

Impact on research: the paper argues that for narrative similarity, calibrating when to spend more compute on a hard instance matters more than adding auxiliary representations to reason about it. The ablation supports this — the symbolic component contributed negligibly end-to-end while confidence-aware routing produced nearly all the gain. It also contributes a clean demonstration that LLM vote distribution behaves as a usable black-box uncertainty signal in a comparative-judgment setting, and it documents conditions under which synthetic-data calibration transfers poorly to human-labeled evaluation.

Real-world applications:

• Content recommendation and discovery, where systems surface stories, films, or books that share theme, plot trajectory, or outcome rather than just keywords.
• Adaption, licensing, and plagiarism review, where reviewers must judge whether two works share a course of action or resolution.
• Search over large fiction or synopsis corpora, where a cheap confident pass can answer most queries and compute can be escalated only for ambiguous ones.
• Educational and editorial tooling that checks whether a summary preserves a source story's structure and ending.

Industry relevance: routing on confidence is directly useful for cost control, since the cascade resolves most cases at a single batched call and spends extra calls only on the 26% that escalate. The paper's negative result about the symbolic fallback is also practically informative for teams weighing whether to invest in hand-crafted narrative features or in better calibration and escalation strategies.

Future Directions

• Improve the symbolic event-chain signal, whose 1% weight likely reflects sparsity from exact verb matching against a 47-word list; the authors suggest fuzzy matching, full narrative event chains, or semantic role labeling.
• Reduce the class decision bias: recall is lower for A-closer cases (68.8%) than B-closer cases (77.6%), and the model predicts B more often than the label distribution warrants, so correcting this bias is named as a direct target.
• Address the development-to-test gap (81.0% to 72.75%), which the authors attribute to possible distribution mismatch, overfitting to synthetic calibration data, or greater difficulty in the final shared-task setting.
• Strengthen reproducibility, since the system depends on a commercial API and stochastic decoding; exact vote distributions can vary across runs and model revisions without pinned model snapshots.
• Prioritize confidence calibration and escalation strategy over additional symbolic feature engineering, given that only 5% of cases reach the symbolic tiebreaker.

Target Audience

Competition participants and practitioners building LLM pipelines who need cost-aware, confidence-based routing; researchers working on uncertainty estimation and self-consistency decoding; and computational narrative researchers interested in how classical narrative theory (Propp, Freytag, Todorov) translates into computable similarity signals — and in evidence about how far those signals actually carry a system in a shared-task setting.

Authors’ abstract

Across self-consistency samples from an LLM, vote agreement tracks instance difficulty: on SemEval-2026 Task 4 (Narrative Story Similarity), supermajority cases (>= 7/8 votes) resolve at 85 percent accuracy, split votes at 67 percent, and perfect ties at 61 percent, a monotone gradient that holds across the development set. We exploit this in CascadeMind, which routes eight Gemini 2.5 Flash votes by consensus, escalates split votes to additional sampling rounds, and falls through to a symbolic ensemble of theory-inspired narrative signals only on perfect ties (5 percent of cases). The system reached 72.75 percent on Track A test, placing 10th of 44 teams. Ablations show that the symbolic component contributes negligibly end-to-end and that nearly all gains come from confidence-aware routing. The takeaway is methodological: for narrative similarity, calibrating when to spend more compute on a hard instance matters more than adding auxiliary representations to reason about it.