TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection

AI summary

Overview

Research area: Audio-based speech and language understanding for telecom fraud detection, with a focus on benchmark construction and evaluation methodology (arXiv:2609.18748v2, cs.SD).

Technical level: Intermediate. The paper assumes familiarity with classifier metrics (Macro-F1, Balanced Accuracy, recall), text-to-speech rendering, automatic speech recognition (ASR), and large language model (LLM) evaluation, but the construction pipeline and the central argument are explained in accessible terms.

Scope: The paper introduces TeleAntiFraud 2.0, a monthly-refreshed, audio-based Chinese telecom-fraud benchmark of 900 calls per frozen snapshot in which fraud calls are contrasted against near-domain lawful calls generated from the same scenario context.

What This Paper Is About

Telecom fraud scripts change quickly and increasingly imitate ordinary customer-service, risk-notification, and verification calls, so it is hard to tell a scam from a legitimate call using keywords or early conversational cues. Existing benchmarks have two problems: fixed test sets cannot absorb scam patterns observed after release, and their non-fraud examples often come from unrelated topics, which lets models exploit topic shortcuts instead of reasoning about the caller's actual actions. The paper builds a benchmark that solves both by generating fraud and lawful calls as siblings under shared scenario context, and by freezing each monthly evaluation set so it can never be overwritten.

Key Contributions

1. A versioned audio benchmark for continuously evolving telecom fraud. Monthly releases incorporate newly observed scam patterns while every published snapshot stays immutable, packaged with the artifacts needed for reproducible and auditable evaluation.

2. The Mixed-Tree Anti-Fraud Generation Pipeline. Fraud case abstracts are converted into mixed dialogue trees and role-matched speech, where fraud and lawful sibling paths share participants, scenario context, opening turns, and early risk language, and diverge only at label-bearing actions.

3. Controlled text and full-set audio evaluations showing that unrelated negatives substantially overestimate detection performance, while near-domain siblings expose false-positive bias, class-prior shortcuts, and prediction collapse across model families.

4. A collapse-aware analysis protocol that complements fraud-class F1 with Macro-F1, Balanced Accuracy, class-conditional recall, and prediction-class distributions.

Main Findings

• Perfect separation disappears with near-domain negatives. Three classifiers (logistic regression, SVM, and a frozen-encoder RoBERTa) reach Macro-F1 of 1.000 against unrelated random negatives and ordinary in-domain negatives (both 1.000 [1.000, 1.000]), but fall to 0.680 [0.643, 0.716], 0.673 [0.639, 0.707], and 0.650 [0.625, 0.675] respectively on near-domain mixed-tree siblings. Mean bigram lexical overlap rises in step, from 0.005 to 0.161 to 0.274.

• Lower linear separability under a shared classifier. Using the same TF–IDF+SVM setup, TeleAntiFraud-28k ASR data reaches Macro-F1 0.9950, Balanced Accuracy 0.9950, fraud recall 0.9900, non-fraud recall 1.0000, and a predicted fraud ratio of 0.4950. TeleAntiFraud 2.0 dialogue data reaches 0.9286 / 0.9420 / 0.9200 / 0.9640 / 0.6253, and TeleAntiFraud 2.0 ASR-test reaches 0.7998 / 0.7758 / 0.9717 / 0.5800 / 0.7878, showing a drop in non-fraud recall and a fraud-biased prediction distribution. The paper notes the cross-benchmark differences mean the gap cannot be attributed to a single construction factor.

• Prediction collapse is widespread across model families. Before prompt-language averaging, 11 of 27 V1 configurations and 15 of 27 deduplicated V2 configurations show an all-FRAUD-like signature: fraud recall approaching 1.0, accuracy approaching the 2:1 fraud prior, and fraud F1 approaching 0.80.

• Snapshot sensitivity is substantial. Table 6 reports fraud F1 / accuracy / fraud recall for each model and input mode on the 900-sample June/V1 and July/V2 sets. Examples: Gemini-Pro/Aud goes from .790/.654/.980 to .733/.582/.899; MiMo-V2.5/Aud from .354/.350/.550 to .663/.550/.670; Seed-2.0-lite/Aud from .007/.003/.335 to .732/.667/.573; Claude-Opus4/ASR from .201/.357/.460 to .800/.667/1.000; Claude-Sonnet4/ASR from .000/.333/.330 to .800/.667/1.000; GLM-5.2/ASR from .012/.337/.335 to .800/.667/1.000. Several systems sit at .800/.667/1.000 on both snapshots (Qwen3-Omni/Aud, DeepSeek-V3/ASR, Qwen2.5-72B/ASR, Qwen3.7-Max/ASR, GPT-4o/ASR on June/V1 at .800/.667/1.000 and July at .800/.666/.999).

• Fraud F1 is class-prior sensitive. Resampling the strict held-out pool at fraud/non-fraud ratios of 1:2, 1:1, and 2:1 over the same five seeds, the all-FRAUD baseline's fraud F1 rises from 0.500 to 0.800 as fraud prevalence increases, while Balanced Accuracy remains 0.500 and non-fraud recall stays zero. TF–IDF+SVM is less sensitive. MiniMax-CN remains fraud-biased even at 1:1, predicting FRAUD for 0.830 of calls and recalling only 0.233 of non-fraud calls.

• Mixed-tree geometry matches the design intent. BGE-small-zh embeddings over 813 dialogues give a mean within-tree distance of 0.0188 versus a cross-tree distance of 0.3351, a 17.8 times ratio. Leave-one-tree-out F1 drops from 0.883 in-domain to 0.605 on held-out trees.

• Pilot quality checks. An 80-dialogue audio package received one completed annotator pass, with mean scores of 4.60 ± 0.70 (dialogue realism), 4.41 ± 0.59 (strategy coherence), 4.35 ± 0.53 (victim reaction plausibility), and 4.13 ± 0.77 (audio naturalness) on a 1–5 scale.

• Label auditing results are cut off in the available text. Table 1 lists a "10-expert audit" for TeleAntiFraud 2.0 and the supplementary section states that ten independent experts each reviewed a 100-item packet (1000 items in total) with expert labels, confidence scores, evidence-sufficiency tags, correctness flags, and notes, but the supplied content ends mid-sentence before the audit outcome is reported.

Methodology in Plain English

The authors build the benchmark in four stages.

First, scenario profiling: online fraud case abstracts are turned into structured profiles that specify the receiver's background, the caller's claimed identity and persuasion strategy, staged objectives, risk-related entities, and risk nodes where the call may escalate, allow verification, or end. Participants and opening context are held fixed across sibling branches, while wording and delivery can vary.

Second, mixed-tree expansion: instead of generating fraud and non-fraud calls separately, the pipeline grows one tree from a shared root and keeps ambiguous, fraud-leaning, non-fraud-leaning, and naturally-terminating continuations together. Each open node proposes at most b high-level plot actions (such as requesting a credential, permitting official verification, refusing, or ending the call); state transitions decide whether a path moves toward fraud or non-fraud. The implemented configuration uses tree depth D = 4 and branching b = 3, giving at most 3^4 = 81 leaves before validation and pruning. A terminal path is labelable only if its trajectory contains sufficient label-bearing evidence.

Third, collaborative dialogue realization with six agents: two role agents for the caller and receiver, and four functional agents for branch expansion, termination, and caller/receiver delivery states. Plot decisions are fixed before any surface wording is generated, so the link between each utterance and its generating action is preserved in the recorded history.

Fourth, speech rendering: validated dialogues are rendered as naturalistic call audio using role-matched voices, with delivery states guiding speaking style and lightweight signal checks removing corrupted outputs.

For evaluation, each monthly snapshot is a pure test set of 900 Chinese calls with a fixed 2:1 fraud-to-non-fraud composition (600 fraud, 300 near-domain non-fraud). Manifests bind each call to its audio, dialogue text, label and rationale, delivery tags, generation and speaker metadata, evaluation prompt, model settings, raw response, parsed prediction, timestamp, and provenance record. Models are run in direct-audio mode or in an ASR+LLM mode where each file is transcribed once with Whisper-medium and the same transcript is reused across LLMs. Prompts in Chinese and English use the same zero-shot task and require one parsed label, FRAUD or NONFRAUD.

Why This Matters

Impact on research. The paper argues that near-domain construction and collapse-aware reporting are core requirements for evaluating audio-based telecom-fraud models under realistic confusable conditions, and that fraud-class F1 alone is insufficient. Its sibling-path design echoes contrast-set evaluation of local decision boundaries, and its immutable-snapshot contract gives the community a reproducible way to absorb new scam patterns without invalidating earlier numbers.

Real-world applications:

• Screening inbound calls at banks and card issuers, where a risk-notification call and a phishing call can share identical opening context and vocabulary.
• Telecom carrier-level fraud filtering and robocall/voice-assistant guardrails that must avoid blocking legitimate service calls.
• Public-security and anti-fraud hotline triage, where analysts need an auditable record of why a call was flagged.
• Model procurement and regression testing: monthly frozen snapshots let an organization check whether a new speech or LLM system has regressed on boundary cases rather than only on average accuracy.

Industry relevance. The benchmark is hosted with monthly snapshots and provenance records, and the frozen manifests record raw responses and parsed predictions, which lets teams audit aggregate scores down to the configuration level without rerunning models. The paper reports the 2024 Global State of Scams estimate of more than USD 1.03 trillion in global scam losses based on 58,329 survey responses, which frames the cost of misjudging the fraud/lawful boundary in both directions.

Future Directions

• Extend the timeline. The paper states that the two current snapshots (June/V1 and July/V2) support snapshot-sensitivity analysis, but that longer-term temporal claims require additional releases. More monthly snapshots would test whether refreshability actually tracks evolving scam patterns.
• Add proper transcription-quality metrics. Clean character error rate (CER) and word error rate (WER) estimates require full-dialogue or segment-level transcription and are explicitly outside the current scoring contract, leaving the ASR transcript artifact usable only as a controlled input for LLM comparison.
• Develop tree-aware training splits. Leave-one-tree-out F1 drops from 0.883 in-domain to 0.605 on held-out trees, which the authors present as motivation for tree-aware splits if the generator is later used for training.
• Broaden independent human validation. The reported dialogue-quality numbers come from a single annotator pilot pass, and the paper excludes additional packet files derived from the first annotator from independent-rating claims. Scaling to fully independent raters, and completing the reported 10-expert label audit across the 100-item packets, would strengthen the artifact-level evidence.

Target Audience

Speech and spoken-language-processing researchers, fraud-detection and anti-fraud machine learning practitioners, and benchmark or dataset builders who care about evaluation design, negative-sample construction, and reproducibility. It is also relevant to product and risk teams at banks, card issuers, and telecom carriers who need to reason about false positives on lawful calls, and to public-security researchers working on telecom fraud because the benchmark is positioned explicitly as a defensive research instrument. Readers without a metrics background will need to look up Macro-F1, Balanced Accuracy, and class-conditional recall to follow the arguments, but the construction pipeline itself is described without heavy mathematics.

Authors’ abstract

Telecom fraud scripts evolve rapidly and are often designed to resemble routine service conversations, creating two key requirements for audio-based telecom-fraud evaluation. First, benchmarks must incorporate newly observed scam patterns without overwriting previously established test sets. Second, they must distinguish fraud from lawful, near-domain calls rather than relying on topic-separated negative examples. We present TeleAntiFraud 2.0, constructed with our Mixed-Tree Anti-Fraud Generation Pipeline and evaluated under a monthly frozen evaluation protocol. The pipeline transforms online fraud-case abstracts into profile-grounded scenarios, expands them through mixed-tree generation, realizes fraud and non-fraud dialogue paths under shared contexts, renders validated dialogues as role-matched speech, and freezes the resulting audio, labels, prompts, manifests, and provenance records for each monthly evaluation set. Each frozen set contains 900 Chinese calls, comprising 600 fraud and 300 near-domain non-fraud cases. Controlled text experiments show that three classifiers achieve perfect macro-averaged F1 (Macro-F1) when evaluated against unrelated or ordinary negatives, but drop to 0.65-0.68 with near-domain sibling negatives. Full-set audio and automatic-speech-recognition plus large-language-model (ASR+LLM) evaluations further reveal class-prior shortcuts, prediction collapse, and snapshot sensitivity. Together, these findings establish near-domain construction and collapse-aware reporting as core requirements for evaluating audio-based telecom-fraud models under realistic confusable conditions. The accompanying research artifact includes the construction code, evaluation scripts, manifests, and documentation. Our dataset and code are available at https://anonymous.4open.science/r/TeleAntiFraud-2_0-EEB2/.