FinRpt: Dataset, Evaluation System and LLM-based Multi-agent Framework for Equity Research Report Generation

AI summary

Overview

Research area: Natural language processing applied to finance, specifically the automatic generation of equity research reports (ERRs) using large language models and multi-agent systems.

Technical level: Intermediate. The paper is readable without deep mathematical background, but familiarity with LLM agents, supervised fine-tuning, and reinforcement learning helps. The financial domain concepts (income statements, balance sheets, cash flows, buy/sell ratings) are explained in sufficient context.

Scope in one sentence: The paper defines the equity research report generation task, releases a Chinese-market dataset of 6,825 report samples with an 11-metric evaluation system, and proposes a nine-agent framework trained with SFT and reinforcement learning that outperforms standalone frontier LLMs on this task.

What This Paper Is About

Equity research reports are the documents professional analysts write to assess a company's financial health, management, risks, and investment potential. Producing one requires gathering scattered information from financial statements, announcements, news, and stock history, then synthesizing it into a coherent multi-section document. No prior work had formalized the automatic generation of these full reports as a task, partly because no public dataset or suitable evaluation metrics existed. This paper fills that gap by building a dataset, defining how to measure quality, and designing a multi-agent system that decomposes the writing job into specialized subtasks.

Key Contributions

1. Task formulation. The authors formally define the Equity Research Report generation task for the first time: given a stock ticker and a research date, the system gathers six types of company-related data and produces a structured report with six required sections. They argue the formulation generalizes to report generation in other domains.

2. FinRpt benchmark and dataset. They build an open-source benchmark comprising a 6,825-sample Chinese ERR dataset (with an English-translated version) covering 800 CSI800 stocks across ten weekly analysis dates, plus an automatic dataset construction pipeline that integrates seven financial data types and three enhancement modules to raise generated reports to near-expert quality.

3. Comprehensive evaluation system. They devise 11 metrics spanning text similarity, numerical richness, prediction accuracy, and finance-specific professionalism (financial numerics, news relevance, company/market/industry insight, investment reasoning, risk coverage, and writing quality), including a position-bias-corrected LLM judge protocol.

4. FinRpt-Gen multi-agent framework. They propose a nine-agent pipeline organized into information extraction, analysis, and prediction modules, and train the four most demanding agents with LoRA-based supervised fine-tuning and DAPO reinforcement learning, achieving the best results across all evaluation categories.

Main Findings

• Multi-agent decomposition beats single LLMs. FinRpt-Gen consistently outperformed standalone models of comparable or larger size. For example, single GPT-4o scored 40.72 ROUGE-L and 48 percent recommendation accuracy, while FinRpt-Gen with GPT-4o reached 48.44 ROUGE-L and 51 percent accuracy.

• Fine-tuning on the constructed dataset surpasses closed-source frontier models. After SFT on FinRpt, open-source models such as Qwen2.5-7B-Instruct climbed to 48.83 ROUGE-L and 54 percent accuracy, exceeding both GPT-4o and Gemini-2.5-Pro in nearly all basic metrics despite being far smaller.

• Reinforcement learning adds a further gain. The final model, FinRpt-Gen with Qwen2.5-7B-Instruct-SFT-RL, achieved 100 percent completion rate, 55 percent recommendation accuracy, 49.06 ROUGE-L, 82.43 BERTScore, and 95.15 percent NumberRate, the best scores in the comparison table.

• Professionalism scores follow the same pattern. Under the LLM-judged metrics, the trained model averaged an adjusted win rate of 0.69 against baselines, compared with 0.59 for FinRpt-Gen with GPT-4o. It beat GPT-4o on financial numerics, news, and company/market/industry criteria, and trailed it only on writing style.

• Generated data approaches expert quality. Three senior financial analysts rated 30 dataset reports against 30 expert-written reports on a 0–5 scale. Averages were 4.20 for FinRpt versus 4.30 for expert reports, with Fleiss' kappa scores between 0.84 and 0.89, indicating strong inter-rater agreement and reliable evaluation.

• Every architectural component matters. Ablation showed removing the finance extraction agents dropped accuracy from 55 to 47 percent and ROUGE-L from 49.06 to 38.93; removing news extraction dropped ROUGE-L to 46.02; collapsing the three analysis agents into a single GPT-4o dropped accuracy to 51 percent.

• The pipeline is fast and cheap. End-to-end report generation takes roughly three to four minutes: 120–160 seconds for data crawling (which can be cached), about 60 seconds for nine agent API calls, and about one second for PDF rendering.

Methodology in Plain English

The authors started by deciding what information an analyst actually needs. They settled on six inputs for each stock and date: company profile data, financial statement indicators, company announcements, company-related news, historical stock prices, and market index history. These come from Yahoo Finance, AKShare, Eastmoney, and Sina Finance.

Using those inputs, they defined a target report structure with six sections: financial analysis, news analysis, management and development analysis, risk analysis, investment potential assessment, and a final buy-or-sell recommendation rating.

To create training data without hiring hundreds of analysts, they had their own multi-agent system draft reports using GPT-4o, then ran those drafts through three cleanup steps. A rating corrector checked whether the generated buy/sell call matched the actual subsequent stock movement, discarding and regenerating reports where it did not. An expert-report corrector retrieved real analyst reports published around the same date and prompted GPT-4o to reconcile accuracy, logic, and style. A final polisher improved readability. This produced the FinRpt dataset.

For the model itself, they assigned four agents to extract information (ranking news by likely stock impact, and pulling figures from income statements, balance sheets, and cash flow statements), four agents to write analysis sections, and one prediction agent to combine everything with price and index history into the investment assessment and rating.

They then fine-tuned four key agents (finance analysis, news analysis, status analysis, and prediction) with LoRA adapters. For the prediction agent specifically, they added a reinforcement learning stage using DAPO, rewarding the model 60 percent for getting the buy/sell direction right and 20 percent each for ROUGE-1 and ROUGE-L overlap on the investment rationale. Evaluation combined automatic text metrics, LLM-as-judge scoring across six professional dimensions with order swapping to cancel position bias, and a human study for validation.

Why This Matters

The work shows that complex, multi-section professional documents can be generated automatically by decomposing them into subtasks and training smaller open models on carefully curated synthetic data. It also demonstrates that task-specific evaluation must go beyond ROUGE and BERTScore: for financial reports, getting a cash flow figure wrong or misjudging a stock direction matters far more than stylistic similarity.

Real-world applications:

• Investment research automation. Brokerages and asset managers could generate draft reports in minutes rather than hours, freeing analysts for deeper judgment calls.
• Retail investor access. Individual investors who cannot afford professional research subscriptions could receive structured company analyses on demand.
• Financial education and training. Junior analysts and finance students could compare their drafts against automatically generated reports structured the way professionals write them.
• Downstream FinTech systems. Reports provide explicit rationales for stock calls, which can feed into portfolio construction, risk monitoring, or algorithmic trading pipelines.

Industry relevance: The two-month data window, weekly cadence, and three-to-four-minute turnaround align with real market reporting cycles. Cost analysis based on OpenAI API pricing suggests the approach is economically viable at scale, though the paper's reliance on commercial APIs for dataset construction is a consideration for firms with data governance constraints.

Future Directions

• Broaden market and language coverage. The dataset covers only Chinese CSI800 stocks over a ten-week window. Extending to other exchanges, longer histories, and additional languages would test whether the pipeline generalizes.
• Improve and verify evaluation further. The LLM-judge metrics could be strengthened with automated numeric fact-checking against source financial statements, addressing the risk that a judge model shares blind spots with the generator.
• Reduce dependence on proprietary models. The dataset construction pipeline uses GPT-4o at multiple stages. Distilling that process into open models would make reproduction cheaper and more transparent.
• Test longer-horizon prediction value. Recommendation accuracy is measured against short-term price trends. Whether these generated reports support profitable decisions over realistic investment horizons remains untested.

Target Audience

This paper suits NLP researchers working on long-form document generation, multi-agent systems, and domain-specific benchmarks; quantitative finance and FinTech practitioners evaluating whether LLMs can automate research workflows; and applied machine learning engineers interested in combining supervised fine-tuning with reinforcement learning for professional writing tasks. Readers seeking a formal treatment of multi-agent orchestration theory will find less here, as the contribution is empirical and benchmark-oriented.

Authors’ abstract

While LLMs have shown great success in financial tasks like stock prediction and question answering, their application in fully automating Equity Research Report generation remains uncharted territory. In this paper, we formulate the Equity Research Report (ERR) Generation task for the first time. To address the data scarcity and the evaluation metrics absence, we present an open-source evaluation benchmark for ERR generation - FinRpt. We frame a Dataset Construction Pipeline that integrates 7 financial data types and produces a high-quality ERR dataset automatically, which could be used for model training and evaluation. We also introduce a comprehensive evaluation system including 11 metrics to assess the generated ERRs. Moreover, we propose a multi-agent framework specifically tailored to address this task, named FinRpt-Gen, and train several LLM-based agents on the proposed datasets using Supervised Fine-Tuning and Reinforcement Learning. Experimental results indicate the data quality and metrics effectiveness of the benchmark FinRpt and the strong performance of FinRpt-Gen, showcasing their potential to drive innovation in the ERR generation field. All code and datasets are publicly available.