PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval

AI summary

Overview

Research area: Financial information retrieval (IR) using large language models; specifically LLM-based document and chunk ranking for SEC filings and financial QA.

Technical level: Intermediate — assumes familiarity with retrieval metrics (NDCG, Recall), prompting paradigms (CoT, ReAct, ToT), in-context learning, and multi-agent systems.

Scope: A training-free framework that empirically determines when prompt engineering, in-context learning, and multi-agent coordination each add value for financial retrieval, benchmarked across the FinAgentBench, FiQA-2018, and FinanceBench datasets.

What This Paper Is About

Financial filings are long, dense, and domain-specific, making it labor-intensive for analysts (and automated systems) to pinpoint relevant documents and the most useful passages within them. PRISM investigates whether off-the-shelf LLMs — with no fine-tuning — can perform this retrieval and ranking reliably, and which combination of prompting, example selection, and multi-agent decomposition actually improves results. The central question is not "can we build something new?" but "which of these rapidly adopted LLM techniques genuinely pays off, and under what conditions?"

Key Contributions

1. A training-free framework (PRISM) that unifies refined system prompting, in-context learning (ICL), and optional multi-agent coordination for both document-level and chunk-level financial ranking, evaluated across three complementary datasets.

2. Systematic ablation studies showing when each component helps: prompt engineering delivers consistent gains cheaply, ICL helps only when scoped selectively (document level, not chunk level), and multi-agent systems benefit mostly larger models.

3. Evidence that simpler beats complex: non-agentic configurations with the strongest prompt (P4) and document-level ICL outperform deeper multi-agent pipelines, particularly on fine-grained chunk ranking tasks where error propagation compounds.

4. First latency, token-usage, and cost analysis for FinAgentBench, providing reproducible baselines for accuracy–efficiency trade-offs in training-free financial IR, including production-readiness considerations.

Main Findings

• Best configuration is non-agentic: Run 19 (P4 prompt, GPT-5 for document ranking, GPT-5 with document-level ICL-5/TE3-S) achieved NDCG@5 of 0.71818 on FinAgentBench, ranking 3rd overall and the only training-free method in the top three — trailing the leader (a fine-tuned embedding model) by just 0.007.
• Prompt wording beats prompt volume: Later, leaner prompts (P3–P4) that encouraged explicit reasoning while cutting verbosity outperformed earlier prompts stuffed with statistical keyword priors. Prompt clarity, not informational density, drove gains.
• ICL must be scoped carefully: Applying ICL only at the document stage consistently improved results; extending it to both document and chunk stages degraded performance through context overload and fragmented attention. In agentic workflows, more shots (10–15) often hurt compared to 5.
• Multi-agent systems are model-size dependent: Smaller models (GPT-4o-mini) suffered heavily in deeper agent graphs (A2 variant dropped to 0.51518), while scaling to GPT-5-mini lifted the A3 workflow from 0.53400 to 0.66291. Coordination overhead outweighs decomposition benefits unless individual agents are strong enough to maintain coherent intermediate reasoning.
• Hybrid agentic/non-agentic works well: Restricting agentic reasoning to document ranking while keeping chunk ranking non-agentic yielded strong scores (0.70685 and 0.69439), confirming that chunk-level agentic control introduces excessive overhead.
• Strong cross-dataset results: On FiQA-2018, PRISM achieved 78.59% improvement in NDCG@10 and 52.24% in Recall@100 over the BM25+Cross-Encoder baseline. On FinanceBench oracle QA, the P1 prompt reached 98% accuracy, surpassing the replicated baseline.
• Practical cost profile: Document ranking averages 10.43s and $0.0155 per query; chunk ranking averages 131.13s and $0.2768 per query — roughly $0.30 per complex query, viable for high-value financial analysis.
• Stable and reproducible: All key runs showed low variance (coefficient of variation below 1.6%), with Welch's t-tests confirming statistically significant improvements over the Run 12 baseline (p < 0.05).
• Task-dependent prompt preference: The prompt variant that won on reranking tasks (P3/P4) was not the one that won on QA (P1), showing optimal prompting depends on whether the task is ranking or factual extraction.

Methodology in Plain English

The researchers began with exploratory data analysis on FinAgentBench, FiQA-2018, and FinanceBench to understand what makes financial retrieval hard — unequal document-type distributions, concentrated keyword vocabularies per filing type, sparse relevance labels, and wide variation in chunk length. These observations directly informed the design of domain-specific prompt priors.

They then built PRISM as three loosely coupled modules:

1. Prompt engineering: Four progressively refined prompt templates (P1–P4) that layer ReAct, Chain-of-Thought, and Tree-of-Thought reasoning scaffolds, with P4 emphasizing streamlined instructions and strict output formats to reduce hallucination.
2. In-context learning: An offline FAISS vector store of training query–document pairs embedded with OpenAI's text-embedding-3 models. At inference, the top-k most semantically similar exemplars are retrieved and prepended to the prompt, dynamically tailoring demonstrations to each query. The exemplar store is kept separate from the candidate corpus to avoid contaminating rankings.
3. Multi-agent system: A LangGraph-based state framework where specialized agents (Question Analyzer, Document Experts, Financial Analyst, Risk Analyst, Evidence Extractor, Contextual Reasoner, Noise Remover, Quick Filter) handle filtering and scoring. Four chunk-ranking architectures (A1–A4) span from a four-agent parallel scoring ensemble to a minimal two-agent configuration.

They ran 34 configurations across four GPT models (GPT-4o-mini, GPT-4.1, GPT-5-mini, GPT-5) and two embedding sizes, measuring NDCG@5 on FinAgentBench's public/private validation splits, NDCG@10 and Recall@100 on FiQA-2018, and answer accuracy on FinanceBench under oracle context. They also logged latency, token usage, and dollar cost, and repeated key runs to check statistical stability.

Why This Matters

Impact on research: The paper challenges the prevailing assumption that more agents and more decomposition equal better performance. By isolating exactly when each component helps, it provides a corrective empirical baseline for the growing body of work on agentic IR. The cost-and-latency benchmarking is also a first for FinAgentBench, enabling future comparisons that account for efficiency, not just accuracy.

Real-world applications:

• Financial analyst workflows: Rapidly locating the right filing (10-K, 10-Q, 8-K, DEF 14A, earnings transcript) for a given research question, then surfacing the specific passages that answer it.
• Regulatory and compliance review: Screening SEC filings for risk disclosures, compensation language, or restatement signals with verifiable passage-level evidence.
• Financial QA systems: Powering RAG pipelines that answer numerical or factual questions grounded in public filings, useful for investor relations, due diligence, and audit support.
• Enterprise search products: Deploying cost-predictable, training-free retrieval without needing in-house fine-tuned embedding models or GPU infrastructure for model training.

Industry relevance: The results suggest that organizations can get competitive financial retrieval performance using off-the-shelf commercial LLM APIs and careful prompt design, without investing in fine-tuned embedding models or complex agent orchestration. The per-query cost of roughly $0.30 is defensible where accuracy is worth more than inference latency, and the framework's training-free nature makes provider swapping straightforward.

Future Directions

• Extending to open-source model families (e.g., Llama, Mistral, Qwen) — explicitly named by the authors as the most important next direction, since all experiments used OpenAI models due to infrastructure constraints.
• Better chunk-level coordination strategies: The paper shows chunk-level agentic reasoning fails due to coordination overhead, but leaves open whether lighter-weight decomposition (e.g., pairwise cascades or tournament ranking) could avoid error propagation.
• Adaptive ICL selection: Current retrieval is purely embedding-similarity based; investigating quality-aware exemplar selection, diversity constraints, or label-space balancing could stabilize the degradation seen with more shots.
• Scalable cost reduction for long-context tasks: Chunk ranking consumes roughly 100K tokens per query with 85% spent on prompts; architecture-level compression or hierarchical summarization could bring costs down without sacrificing accuracy.
• Longitudinal and cross-market validation: Testing on filings beyond 2023–2024 SEC data, and on non-US regulatory documents, would clarify how much the domain priors depend on US filing conventions.

Target Audience

Financial NLP and information retrieval researchers; practitioners building RAG or enterprise search systems over financial documents; engineering teams evaluating whether to invest in fine-tuned models versus prompt-and-ICL pipelines; and product managers in fintech, compliance, or investment research who need realistic latency and cost numbers before committing to an LLM-based retrieval architecture. The paper is most valuable to readers who already understand ranking metrics and LLM prompting, but the empirical guidance on "simpler configurations often win" is broadly accessible.

Authors’ abstract

With the rapid progress of large language models (LLMs), financial information retrieval has become a critical industrial application. Extracting task-relevant information from lengthy financial filings is essential for both operational and analytical decision-making. We present PRISM, a training-free framework that integrates refined system prompting, in-context learning (ICL), and lightweight multi-agent coordination for document and chunk ranking tasks. Our primary contribution is a systematic empirical study of when each component provides value: prompt engineering delivers consistent performance with minimal overhead, ICL enhances reasoning for complex queries when applied selectively, and multi-agent systems show potential primarily with larger models and careful architectural design. Extensive ablation studies across FinAgentBench, FiQA-2018, and FinanceBench reveal that simpler configurations often outperform complex multi-agent pipelines, providing practical guidance for practitioners. Our best configuration achieves an NDCG@5 of 0.71818 on FinAgentBench, ranking third while being the only training-free approach in the top three. We provide comprehensive feasibility analyses covering latency, token usage, and cost trade-offs to support deployment decisions. The source code is released at https://bit.ly/prism-ailens.