Credit Cards, Confusion, Computation, and Consequences: What Can We Uncover About Language Model Reasoning?

AI summary

Overview

Research area: Natural Language Processing / financial numerical reasoning and benchmark construction (arXiv:2607.26952v1 [cs.CL]).

Technical level: Intermediate. The paper is readable without deep finance background, but it assumes familiarity with prompting methods (Chain-of-Thought, Program-of-Thought) and logistic regression.

Scope: The paper introduces a benchmark of 1,800 numerical reasoning questions derived from 27 real credit card agreements, evaluates 11 language and reasoning models on it, and analyzes why models fail.

What This Paper Is About

People in the United States owe $1.2 trillion in credit card debt, and financial literacy has stayed near 50% for eight consecutive years according to the survey evidence cited in the paper. Existing financial reasoning benchmarks focus on stock markets, investment analysis, firm performance, and professional finance tasks, not on the everyday personal finance questions consumers actually ask. The authors build the first benchmark for financial literacy numerical reasoning grounded in real credit card agreements, then measure how well language models answer those questions and where their reasoning breaks down.

Key Contributions

1. A new benchmark grounded in real contracts. The authors release a dataset of 1,800 question–answer pairs derived from 27 credit card agreements collected from the Consumer Financial Protection Bureau (CFPB) database, spanning major issuers such as American Express and Discover, plus Barclays for European coverage. Agreements were drawn from Q4 2024 and chosen by market share; the issuers included collectively account for over 80% of total outstanding balances. The dataset is referred to in the paper as CreditCardQA, CreditQA, and FinLitQA in different sections.

2. First-person question variants. Of the 1,800 questions, 1,308 are third-person and 492 are first-person, reflecting how consumers phrase questions about their own fees, interest, and payments.

3. A systematic evaluation of 11 models under two prompting strategies. Eight open-weight models (DeepSeek-R1, Qwen-QwQ, GPT-OSS-20B, GPT-OSS-120B, Kimi-K2-Thinking, Llama-3.3-70B, Mistral-Small-24B, GLM-4.6) and three closed-source models (GPT-5, GPT-5-mini, Gemini 3.0 Pro) are evaluated under Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting, with accuracy reported at ±0.2% and ±5% relative error tolerances.

4. Error and difficulty analysis. A manual audit of 50 incorrect GPT-OSS-120B responses classifies five non-mutually-exclusive failure modes, and a logistic regression over GPT-OSS-120B responses isolates which structural, linguistic, and input features predict correctness.

Main Findings

• PoT beats CoT for every model. At the ±5% tolerance, PoT improves accuracy over CoT for all 11 models, with gains ranging from 0.9% to 6.2%. The largest gains go to models with weaker CoT baselines (Llama 3.3: +6.2; Qwen-QwQ: +6.1; DeepSeek-R1: +5.7), while stronger CoT baselines gain less (GPT-5-mini: +0.9; GPT-OSS 20B: +1.2).

• Top performers. GPT-OSS 120B, Gemini 3.0 Pro, and GPT-5 are the top three models across all evaluated settings. Gemini 3.0 Pro and GPT-5 reach accuracy close to 80% at the ±5% tolerance, and the open-weight GPT-OSS 120B slightly surpasses them under PoT (79.2% at ±5%, 72.0% at ±0.2%).

• The LLM/LRM gap depends on prompting. Under CoT, large reasoning models outperform large language models by roughly 4–5 percentage points at the ±5% tolerance; under PoT that gap shrinks to roughly 2–3 percentage points.

• PoT gains are not monotonic in model size. The 32B Qwen-QwQ benefits more from PoT than the larger GPT-OSS 120B, and the 671B DeepSeek-R1 benefits comparably to Qwen-QwQ. Within families, larger models still score higher (GPT-5 over GPT-5-mini; GPT-OSS 120B over GPT-OSS 20B).

• Failures come from rules, not arithmetic. In the 50-question GPT-OSS-120B audit, formula application (concept substitution) was the most prevalent error at 70% (35 of 50), followed by missing condition application at 54% (27), additional unnecessary steps at 18% (9), numerical calculation error at 14% (7), and misunderstanding of the problem at 8% (4). Numerical errors were mostly arithmetic or calendar mistakes, such as miscounting days between dates or assuming 30 versus 31 days in a month.

• Comparisons are the single biggest difficulty driver. In the regression, comparison-based reasoning reduces the odds of a correct answer by approximately 74% (β = −1.33, p < 0.001), the largest effect among all features. Holding other factors constant, removing comparison operations would increase the odds of correctness by nearly 4 times.

• More computational steps correlate with higher accuracy. The number of required steps is positively associated with correctness (β = 0.33, p = 0.012), about a 39% increase in odds per additional step. The authors attribute this to multi-step structure forcing explicit application of contractual conditions rather than to longer questions being inherently easier.

• First-person framing helps. Third-person phrasing corresponds to an approximate 42% reduction in the odds of a correct response (β = −0.55, p = 0.005). The authors suggest first-person queries more closely match the instruction-tuning distribution.

• Monetary values and conditional branching hurt. Explicit monetary quantities reduce the odds of a correct answer (β = −0.50, p = 0.004), and conditional branching reduces performance (β = −0.59, p = 0.046), after controlling for reasoning structure and numerical operations.

• Agreement-specific difficulty varies. All agreement-level effects are negative, with statistically meaningful differences clustering by card type: cards designed for limited credit, store-branded retail cards, and rewards or brand-affiliated cards.

• The agreements are hard to read. Flesch–Kincaid readability scores for the source agreements have a minimum of 9.2 and an average of 11.3, corresponding to reading levels typically associated with college graduates or professionals.

• Dataset complexity. The dataset contains 821 unique operator sequences and 602 unique input–output combinations. Median/mean figures for the underlying agreements include 9,115/8,830 words, 20/22.6 percentages, 15.5/19.25 dollar amounts, and 55.5/68.6 bullet points. Questions have median/mean length of 33/36.7 words and require median/mean 3/4.4 math operations.

Methodology in Plain English

The team collected 27 credit card agreements from the CFPB database (Q4 2024), including American Express, Discover, and Barclays, and selected issuers by market share across three product categories per issuer: premium/elite cards with annual fees, everyday retail cards, and subprime cards with no annual fees. Because the CFPB files are PDFs with images and tables, the researchers converted them to Markdown using Marker and manually verified quality, then generated questions using only the Markdown so no outside information leaked in.

Question creation ran for approximately ten months and was done by annotators with finance and machine learning expertise — PhD holders, machine learning for finance researchers, a Master's in Quantitative Finance holder, CFA-level contributors, and people with hedge fund, investment banking, and asset management experience — under a Chair Professor of Finance. Annotators read each agreement in full, identified financially relevant numeric terms (values with $ or % signs, following Shah et al. 2024), and wrote questions from those terms. This yielded 280 extracted terms and 1,800 questions.

Questions were split into a development set of 800 with full annotations released publicly and a test set of 1,000 held out for a leaderboard. Models were prompted with CoT and PoT, answers were extracted using a pipeline similar to Zhao et al. (2024b) with manual verification for categorical outputs, and accuracy was scored at ±0.2% and ±5% relative error tolerances. The authors also note they evaluated a simple retrieval-augmented generation (RAG) setting, with details placed in Appendix J of the paper (not reproduced in the excerpt above).

For the error analysis, two authors manually reviewed 50 incorrect responses from GPT-OSS-120B, described as a focused audit of an accessible model rather than a characterization of all 11 models. For the difficulty analysis, they fit a logistic regression predicting whether GPT-OSS-120B answered a question correctly, with features for structural reasoning properties (total steps, unique operators, conditional branching, comparisons, division, max/min, rounding, exponentiation), linguistic framing (third-person vs. first-person), input unit types (monetary, percentage, temporal), contextual features (number of financial terms, hardship clauses), and a per-agreement baseline.

Why This Matters

Impact on research. The paper argues that prior financial reasoning benchmarks target stock market tasks rather than the personal finance decisions that affect the largest population. It also contributes a broader finding about evaluation design: that first-person versus third-person phrasing shifts measured accuracy, which suggests benchmarks written in formal third-person style may not reflect how models perform on natural user queries. The authors connect this to RLHF optimization for assistant-like interactions and to work on alignment faking showing that models respond to subtle contextual cues.

Real-world applications:

• Consumer-facing financial assistants that answer questions about minimum payments, penalty APRs, and late fees at the moment of decision.
• Automated explanation of credit card agreement terms for cardholders who cannot read multi-page contracts at a college-graduate reading level.
• Developer tooling that favors symbolic reasoning frameworks such as PoT, which the authors recommend as a way to narrow the logic-processing bottleneck.
• Regulatory disclosure design, where the authors suggest policymakers consider standards for "LLM-accessible" disclosures so automated tools give consumers more accurate guidance.

Industry relevance. The stakes are asymmetric. The authors emphasize that errors concentrate in edge cases like late-payment penalties and small-balance scenarios, which are more likely to affect lower-income or financially vulnerable individuals. The paper also notes that financial literacy and access to effective financial education remain lower among vulnerable and historically underserved populations, and cites evidence that women tend to start with lower credit limits than men when entering the credit card market. For model developers, the finding that open-weight GPT-OSS 120B matches or slightly exceeds closed-source systems under PoT is relevant to deployment decisions.

Future Directions

• Extending the benchmark. The authors state that their more lenient 800/1,000 dev/test split is intended to leave sufficient data for analysis and to facilitate future dataset extensions using the same methodology.
• Improving domain-specific financial reasoning. Because the dominant error is applying the correct concept with the wrong formula or contractual rule, the open question is how to give models better grounding in agreement-specific conditional logic rather than general financial knowledge.
• Investigating linguistic framing systematically. The 42% odds reduction for third-person phrasing raises the question of how other framing dimensions — beyond person — affect measured model accuracy in high-stakes evaluations.
• Characterizing frontier models. The error analysis is explicitly limited to GPT-OSS-120B as an accessible model rather than all 11 evaluated models, leaving a focused audit of closed frontier systems as an open task.
• Testing retrieval augmentation. The paper reports a simple RAG setting but places its implementation and results in Appendix J, which is not detailed in the portion of the paper available here.

Target Audience

Researchers building or evaluating reasoning benchmarks who need a high-stakes, condition-heavy testbed beyond math and stock market tasks; NLP practitioners working on prompting strategies such as CoT and PoT; financial technology developers building consumer-facing financial assistants; and policymakers or consumer-advocacy researchers interested in whether AI tools can reliably explain credit card terms to financially vulnerable users. The paper is most useful to readers who already understand prompting terminology, since the core value lies in the benchmark design, the error taxonomy, and the regression-based difficulty analysis rather than in introducing new modeling techniques.

Note on naming and reporting: the paper uses the names CreditCardQA, CreditQA, and FinLitQA for its dataset in different places, and the license is stated as CC BY 4.0 on the arXiv listing but as CC-BY-NC-SA 4.0 in the ethics statement. The excerpt provided does not include the appendices referenced for the question-creation steps (Appendix B/C), prompt details (Appendix F), verification pipeline (Appendix G), term definitions (Appendix H), full regression coefficients (Appendix I), or RAG results (Appendix J), so those details are not reported here.

Authors’ abstract

We introduce CreditCardQA, the first financial literacy benchmark for numerical reasoning derived from real credit card agreements. The dataset contains 1,800 questions, including first-person variants that reflect how consumers naturally ask about fees, interest, and payments. We evaluate a range of large language and reasoning models under Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting. Overall, PoT yields consistent performance gains, particularly for models with weaker baseline reasoning, and narrows gaps between open- and closed-source systems. Through error analysis, we show that failures arise less from arithmetic and more from misapplied financial rules, missed conditions, and misunderstandings of contractual terms. We further analyze question difficulty and find that comparisons, conditional logic, and monetary constraints are especially challenging. We also find that errors often arise in edge cases such as late-payment penalties or small-balance scenarios that are more likely to affect lower-income or financially vulnerable individuals.