FRAUDSkill: Structured Frozen-Weight Skill Optimization for Audio Anti-Fraud Detection

AI summary

Overview

Research area: Audio anti-fraud detection — specifically, adapting frozen large audio-language models to make structured, closed-set decisions about telecom scam calls.

Technical level: Advanced. The paper assumes familiarity with large audio-language models, prompt/program optimization, beam search over candidate artifacts, and class-balanced evaluation metrics.

Scope: The paper proposes FRAUDSkill, a framework that leaves a frozen audio-language model (Qwen2-Audio-7B-Instruct) untouched and instead optimizes an external layer of skill programs, routing policies, and decision rules to convert open-ended speech understanding into protocol-compliant fraud decisions on the TeleAntiFraud benchmark.

What This Paper Is About

Telecom fraud detection from spoken audio is not an open-ended comprehension task — deployed systems must return answers from a fixed label ontology through a chained protocol: identify the service scenario, decide whether the call is fraudulent, and only then assign a fraud type. Fine-tuning encodes these rules into model weights, which must be re-optimized whenever fraud patterns or labeling policies change; prompt engineering is flexible but hard to maintain and offers weak guarantees that outputs stay inside the label set or remain consistent across the chain. FRAUDSkill's goal is to make a completely frozen audio-language model produce valid, consistent, closed-set anti-fraud decisions by optimizing only external, inspectable artifacts.

Key Contributions

1. Problem formulation. The authors cast audio anti-fraud detection as a structured frozen-weight adaptation problem, in which a frozen audio-language model must support controllable closed-set predictions and chained decisions over scene, fraud status, and fraud type, rather than open-ended generation.

2. The FRAUDSkill framework. It combines route-aware external skill optimization (a root instruction, a named skill library, and a route policy that selects skills per decision step) with structured multi-path inference (label projection, route normalization, complementary frozen-actor trajectories, and a validation-fitted class-balanced selector).

3. Empirical results on TeleAntiFraud. FRAUDSkill reaches 73.50% Macro-F1, outperforming the shared frozen-model baseline by 31.96 percentage points while reducing the invalid-output rate to 1.94%.

4. Isolation of component contributions. Ablations and error analyses separate the value of textual skill search from the value of structured output control, showing where the gains actually come from.

Main Findings

• Complete system performance: FRAUDSkill achieves 73.50% Macro-F1, 79.40% W-F1, 78.72% Accuracy, 58.87% Joint Accuracy, and a 1.94% Invalid Rate on the complete TeleAntiFraud test set.

• Baseline comparison: The shared frozen-model baseline scores 41.54% Macro-F1, 35.92% W-F1, 32.87% Accuracy, 7.10% Joint Accuracy, and a 36.04% Invalid Rate. FRAUDSkill's 31.96-point Macro-F1 gain is an absolute difference.

• Generic skill optimization methods did not help: SkillOpt reached 37.67% Macro-F1 (−3.87 versus baseline) with a 29.00% Invalid Rate; EvoSkill reached 39.07% (−2.47) with a 34.19% Invalid Rate. Both lowered invalid outputs relative to the 36.04% baseline but degraded class-balanced discrimination.

• Text-only route-aware search gave a marginal gain: FRAUDSkill-Text averaged 42.42% Macro-F1 (+0.88 over baseline), 38.87% W-F1, 34.88% Accuracy, 8.18% Joint Accuracy, and a 34.48% Invalid Rate across seeds 42, 43, and 44.

• Text search was unstable: Across the three seeds, FRAUDSkill-Text's sample standard deviation was 1.39 Macro-F1 points — larger than its 0.88-point mean gain over the baseline. Per-seed Macro-F1 was 43.66 (seed 42), 42.69 (seed 43), and 40.92 (seed 44).

• Parameter-updating references: Supervised fine-tuning (SFT) scored 66.06% Macro-F1 and SFT+Memory scored 75.51% Macro-F1 as contextual references. The paper reports these solely as parameter-training reference points, not as frozen-skill baselines.

• Structured components drove the gain: Starting from the best text-layer seed at 43.66% Macro-F1, closed-set projection raised Macro-F1 to 54.47%, and route normalization further raised it to 66.21%. Together these two steps account for 22.55 percentage points.

• Multi-path inference and selection added more: Complementary multi-path inference raised Macro-F1 from 66.21% to 70.31%; reliability weighting raised it to 70.84%, outperforming majority voting by 0.53 percentage points; class-balanced selection produced the final 73.50%, an additional gain of 2.66 points. The structured components contribute 29.84 Macro-F1 points over the best textual program.

• Errors concentrate in the scene route and type route: For FRAUDSkill-Text, the scene-level error rate was 74.2%, with missing or off-ontology outputs accounting for 91.1% of those errors. The fraud route had a 32.9% error rate, of which 60.0% were false-normal decisions that block downstream type classification. The type route had an 83.1% error rate on annotated examples.

• Error counts on the test set (Table 2): 2,677 scene-route examples with 1,985 errors, 1,808 of them missing/off-ontology; 2,677 fraud-route examples with 881 errors, split into 529 false-normal and 340 false-fraud; 1,453 annotated type examples with 1,208 wrong.

• Text-level repairs alone are insufficient: Validation-set repair variants produced Macro-F1 of 41.67 (original), 40.14 (strict scene), 40.26 (type boundary), and 41.18 (balanced) — validity and joint accuracy improved, but the structured-inference gains did not follow from text edits.

• Case study: On a "bank low-interest loan impersonation" call, the frozen model without FRAUDSkill produced non-ontology labels such as "Loan service" and "fake loan fraud," mapped to INV, leaving only one valid prediction out of three. With FRAUDSkill the same frozen actor returned customer consultation, fraud, and bank fraud — all three valid — alongside more specific rationales.

• Evaluation protocol differences: Results are reported at the audio level. After audio-level deduplication, the official SFT split contains 10,711 training samples and 2,677 test samples, of which 1,453 have fraud-type annotations. The original dataset paper used 7,021 interaction-level records, so the two protocols are not directly comparable.

Methodology in Plain English

The authors treat the audio-language model as a black box that reads audio and writes free-form text. Everything that can be changed lives outside it, in an "external skill program" made of three parts: a root instruction that defines the model's role, output contract, and the official label ontology; a library of named skills holding fraud cues, audio evidence, fraud-type boundaries, and cross-route consistency rules; and a route policy that decides which skills are active at each of the three decision steps (scene, then fraud, then type).

Adaptation happens in two strictly separated stages. In offline skill optimization, the frozen model is run over labeled development examples and its full trajectory is recorded — raw responses, parsed labels, route histories, and skipped decisions. An error analyzer compares each trajectory against the gold labels and distinguishes wrong official labels, unmappable outputs, incorrectly skipped type decisions, cross-route inconsistencies, and systematic minority-class errors. A critic model then writes a natural-language diagnosis of the recurring failure modes, and an editor produces a bounded set of revised programs (a branch factor of 3). Starting from a shared initial program, candidates are pooled, scored on a held-out split using a route-aware criterion, and the top candidates are kept in a beam (width 3) for five rounds. The best single program defines FRAUDSkill-Text; the top L programs form the retained set for the full system.

At deployment everything is frozen: the retained programs, label maps, route rules, and selector. Each retained program independently drives the same frozen actor through the ordered routes. Two deterministic post-processing steps run after every route: label projection maps a raw response to an official label via a fixed alias map, marking it INV if no valid mapping exists; route normalization enforces the conditional protocol — if the fraud decision is normal, the type route is not executed and is set to NA, and if the fraud decision is INV the downstream type decision is also treated as invalid. Normalized decisions are appended to the history so each subsequent route sees canonical labels rather than free text. Finally, a selector fitted on a calibration split disjoint from both the test data and the textual-error-feedback examples aggregates the L trajectories using a prespecified family of reliability-weighting and class-balancing configurations, maximizing task-averaged Macro-F1, and a final projection enforces the feasible output set.

Why This Matters

Impact on research. The paper shows that for closed-set, chained decision tasks, the bottleneck for frozen models may not be semantic understanding but output validity and cross-route consistency. It positions external skill optimization as a middle path between parameter updates and hand-maintained prompts, and it introduces diagnostics (trajectory-level error records, invalid-rate metrics, joint accuracy) that make the failure modes of frozen adaptation measurable.

Real-world applications:

• Telecom carriers and regulators deploying scam-call screening that must emit outputs matching a fixed label ontology rather than free-text judgments.
• Fraud-desk triage systems that need an auditable chain — scenario, fraud flag, fraud type — with each step's rationale inspectable so errors can be traced.
• Public-safety anti-fraud units that must update detection policy as new fraud patterns appear, and would otherwise need to retrain or re-prompt a model each time.
• Any deployment where audio or speech evidence must be classified into a regulated label space under an enforced decision protocol (for example, insurance or banking call review), where the "no invalid outputs" constraint matters more than open-ended reasoning quality.

Industry relevance. It offers a training-free, plug-and-play path to deploying a frozen audio-language model. Because task knowledge, label constraints, and decision policy live in external artifacts, they can be revised, inspected, and replaced without touching model weights — which matters when policy changes are frequent and when model retraining is costly or contractually restricted. The paper also reports that full reruns require the TeleAntiFraud audio split, the frozen model, and the critic/editor interfaces, while raw audio and model weights are not redistributed.

Future Directions

• Stabilizing text-layer search. FRAUDSkill-Text's seed-to-seed variation (1.39 Macro-F1 sample standard deviation) exceeds its mean gain over the baseline, so reducing search variance in the textual program layer remains open.

• Closing the gap to parameter-updating references. The complete system is "clearly above the plain SFT reference and slightly below SFT+Memory" (73.50 versus 66.06 and 75.51 Macro-F1 respectively). The paper explicitly notes these comparisons are protocol-bound and not a task-independent ranking; whether the remaining gap can be closed without weight updates is unresolved.

• Improving scene and type routes. The scene route's 74.2% error rate (91.1% of errors being missing/off-ontology) and the type route's 83.1% error rate point to the scene ontology and type boundaries as the largest remaining sources of error.

• Extending beyond the studied protocol. The framework's route-aware design is tied to a three-stage chain on one benchmark with a fixed actor (Qwen2-Audio-7B-Instruct). Whether route-aware skill optimization transfers to other decision-chain lengths, other audio-language actors, or other closed-set audio classification settings is not reported.

Target Audience

Researchers and engineers working on speech and audio-language models, multimodal fraud detection, and frozen-weight or prompt/program optimization; practitioners building anti-fraud or compliance systems that must emit outputs from a fixed label ontology; and readers interested in how much of a structured decision task can be solved by external, inspectable artifacts rather than by training the underlying model. Familiarity with LLM program optimization and class-balanced evaluation metrics is assumed.

Authors’ abstract

Large audio-language models have shown promise for anti-fraud detection by directly processing speech and reasoning over fraud-related evidence. Their deployment, however, requires predictions to follow a predefined label space and a structured decision protocol consisting of service-scenario identification, fraud detection, and conditional fraud-type classification. Existing fine-tuning and prompt-based approaches typically encode task knowledge, constraints, and decision rules into model parameters or manually maintained prompts, making them difficult to adapt as fraud patterns and labeling policies evolve. To this end, we propose FRAUDSkill, a structured frozen-weight adaptation framework that leaves the underlying audio-language model unchanged while optimizing an external layer of skill programs, route-specific policies, and decision rules. We further combine structured output control with validation-guided multi-path inference to ensure protocol-compliant predictions. On the TeleAntiFraud benchmark, FRAUDSkill achieves 73.50% Macro-F1, outperforming the shared frozen-model baseline by 31.96% while reducing invalid outputs to 1.94%. Extensive experiments demonstrate that external skill optimization provides an effective and adaptable solution for structured audio anti-fraud detection without modifying the underlying model. The source code is available at https://anonymous.4open.science/r/FRAUDSKILL-114514.