Self-Verification is All You Need To Pass The Japanese Bar Examination

AI summary

Overview

• Research area: Natural language processing / legal reasoning with large language models, specifically benchmark evaluation on the Japanese bar examination (司法試験).
• Technical level: Intermediate. The paper assumes familiarity with supervised fine-tuning, prompting strategies, exact-match versus partial-credit scoring, and multi-agent LLM pipelines.
• Scope: The paper constructs a format-faithful dataset of Japanese bar exam multiple-choice questions, fine-tunes GPT-4.1 on it with a self-verification inference step, and compares this against base models, a decomposition-based baseline (JBE-QA), and multi-agent pipelines under the exam's official scoring scheme.

What This Paper Is About

Prior work on the Japanese bar examination reformulated exam questions into independent true/false statements, which changes the structure of the exam and leaves open whether models can actually handle real exam items. The paper instead builds a dataset that preserves the original multi-proposition question format and explicit answer formats, then trains a single model that both answers and verifies its own answer. The goal is to test whether a format-faithful, single-model approach can exceed the official passing score on the authentic exam scale.

Key Contributions

1. A format-faithful dataset. The authors collect actual multiple-choice (短答式) exam questions from the Japanese Ministry of Justice spanning 6 years (2019–2024), splitting earlier years (2019–2023, R1–R5) for training and 2024 (Reiwa 6 or R6) for testing. Each instance is a complete exam question with all constituent statements and the original answer choices, and is annotated with subject category (constitutional law 憲法, civil law 民法, criminal law 刑法), year, and awarded points. The dataset contains 460 questions, in contrast to the 2,770 questions of JBE-QA.

2. Self-verification with a shared model. The method fine-tunes the model to produce the exam-style answer directly, then at inference has the same model re-evaluate its own prediction under a different, verification-oriented prompt. Formally, the final answer is a-hat = g_θ(q, f_θ(q)), where f_θ and g_θ share parameters but use distinct prompts. The procedure adds only a single additional forward pass.

3. Evaluation under the authentic exam scale and grading rules. Models are scored with both exact-match accuracy and the official point scheme with partial credit, across the three law sections plus an overall maximum of 175 points.

4. A systematic comparison against alternatives. The paper evaluates base zero-shot and few-shot models, a JBE-QA fine-tuned model, a decomposition-based fine-tune, and two multi-agent configurations (shared-model and separately fine-tuned agents), all under identical prompts and the real exam format.

Main Findings

• First reported pass under the original format. The abstract states the model exceeds the official passing score evaluated on the actual exam scale, described as the first demonstration of an LLM passing the Japanese bar examination without altering the original question structure or scoring rules. The introduction reports a score of 96 on the 2024 Japanese bar exam, whose passing score is 93.
• Best configuration: fine-tuned on the authors' dataset plus self-verification. Accuracy 0.4935, exam scale 94.7 average with a range of 94 to 96; subject averages 22.3 (constitutional law), 42.3 (civil law), 30.0 (criminal law).
• Fine-tuning on the authors' dataset alone already lands near the passing score. Accuracy 0.4675, exam scale 92.3 (min 91, max 93); subject averages 20.3, 42.0, 30.0.
• Base models perform poorly. Zero-shot: accuracy 0.4036, exam scale 67.0 (min 65, max 68). Few-shot with 5 demonstrations: accuracy 0.3896, exam scale 68.3 (min 63, max 71); the paper notes few-shot hardly boosts performance over zero-shot.
• Self-verification helps every model tested. Base few-shot rises from 68.3 to 76.3 average (accuracy 0.4156), and the JBE-QA fine-tune rises from 64.0 to 80.7 average (accuracy 0.4226). The authors describe self-verification as an efficient, model-agnostic technique.
• JBE-QA fine-tuning underperforms despite far more training instances. Accuracy 0.3766, exam scale 64.0 (min 62, max 66); subject averages 8.0, 30.0, 26.0 — lower than the base zero-shot model. The paper attributes this to the task-distribution shift from binary classification and the lack of mechanisms for re-composition at inference time.
• Multi-agent pipelines fail to match a single fine-tuned model. Shared-model multi-agent: accuracy 0.4026, exam scale 75.7 (min 74, max 79). Separately fine-tuned agents: accuracy 0.3969, exam scale 71.0 (min 66, max 77). Both are well below the 93 passing score, and increased agent diversity degraded performance further.
• Qualitative examples illustrate the format gap. In Table 4, the authors' model with self-verification reproduces the exact ground-truth outputs "211" (3 points), "2" (2 points), and "22121" (4 points), while several other configurations receive 0 points or output strings with an incorrect number of digits (for example "1112" or "21222").
• Errors are framed as consistency failures, not knowledge gaps. The authors argue that self-verification corrects local inconsistencies such as a single misjudged statement inside an otherwise correct composite answer, and that format-specific fine-tuning and self-verification act as catalysts eliciting latent knowledge.
• Scoring context. The exam awards 50 points for constitutional law, 75 for civil law, and 50 for criminal law, totaling 175 maximum points. The R6 passing score was 93, with an additional requirement that at least 40% of the points be achieved in each law section. Under the partial-credit rule, when 3 or more questions are grouped with n points, one wrong answer yields n−2 points and two or more wrong answers yield 0 points, so accuracy above 50% can still result in zero points.

Methodology in Plain English

The authors gather real past bar exam questions and keep them exactly as they appear — each question contains several statements (ア, イ, ウ and so on) that must be judged together, and the answer is either a choice number or a concatenated string of digits such as "112". They then fine-tune GPT-4.1 to output that answer directly from the full question, without breaking the question into simpler pieces. At test time, the same fine-tuned model is given a second prompt that asks it to review the question together with its own previous answer and produce a final answer, changing it only if it is clearly wrong. Prompts are calibrated so that the model outputs only the answer, with no explanations or symbols, which removes the need for output-normalization schemes, and the verification prompt explicitly instructs the model to keep the original answer unless it detects a clear inconsistency. For comparison, the authors also fine-tune separate GPT-4.1 models on their dataset and on JBE-QA, run zero-shot and few-shot baselines with 5 demonstrations, and build a four-agent pipeline (retriever, verifier, knowledge extraction, final reasoning) in both a shared-model and an independently fine-tuned configuration. All experiments use identical prompts and are repeated 3 times, and results are reported both as exact-match accuracy and as the official exam score with partial credit.

Why This Matters

Impact on research. The paper argues that benchmarks built by decomposing real tasks into simpler true/false judgments may not transfer to the authentic task, and that preserving question format during fine-tuning and verifying consistency at inference time matter more than adding architectural complexity or external supervision. It also cautions against drawing conclusions about competence from benchmarks that simplify structure and scoring criteria.

Real-world applications.

• Evaluating legal LLM systems under authentic, high-stakes examination conditions rather than simplified proxies.
• Designing professional certification and licensing evaluations where strict output formats and joint constraints determine scoring.
• Building lighter-weight, single-model reliability checks (self-verification) for settings where multi-agent inference cost is prohibitive.
• Informing dataset construction for other domain exams that require combined judgments across multiple propositions.

Industry relevance. The result is relevant to legal-tech and compliance tooling, to providers of foundation models targeting professional domains, and to organizations deciding between elaborate multi-agent orchestration and simpler fine-tuned single models. The authors note that the gains rely on access to a strong pretrained model and assume substantial prior knowledge in the base model.

Future Directions

• Extending the approach to the free-response (論文式) portion of the exam, which requires structured legal argumentation, citation control, and extended reasoning rather than discrete answer selection.
• Testing whether the format-faithful fine-tuning plus self-verification recipe transfers beyond the Japanese bar exam to other professional and licensing exams.
• Investigating whether the approach generalizes to weaker or domain-mismatched base models, given the stated dependence on a strong pretrained model.
• Understanding and mitigating cases where the model's internal legal knowledge is itself incorrect or outdated, which self-verification does not correct.
• Further research into why multi-agent pipelines degrade under tightly constrained, globally consistent tasks, and whether shared representations are a necessary condition for effective agentic behavior.

Target Audience

Researchers and practitioners working on legal NLP, LLM evaluation, and professional-domain benchmarks; engineers deciding between fine-tuning and multi-agent architectures; and anyone designing high-stakes evaluations where answer format and scoring rules are part of the task. Readers interested in benchmark methodology — particularly the difference between decomposed supervision and format-faithful supervision — will find the comparison in Table 1 and Table 3 especially relevant. The paper assumes some background in LLM fine-tuning and prompting, making it most suitable for an intermediate audience.

Authors’ abstract

Despite rapid advances in large language models (LLMs), achieving reliable performance on highly professional and structured examinations remains a significant challenge. The Japanese bar examination is a particularly demanding benchmark, requiring not only advanced legal reasoning but also strict adherence to complex answer formats that involve joint evaluation of multiple propositions. While recent studies have reported improvements by decomposing such questions into simpler true--false judgments, these approaches have not been systematically evaluated under the original exam format and scoring scheme, leaving open the question of whether they truly capture exam-level competence. In this paper, we present a self-verification model trained on a newly constructed dataset that faithfully replicates the authentic format and evaluation scale of the exam. Our model is able to exceed the official passing score when evaluated on the actual exam scale, marking the first demonstration, to our knowledge, of an LLM passing the Japanese bar examination without altering its original question structure or scoring rules. We further conduct extensive comparisons with alternative strategies, including multi-agent inference and decomposition-based supervision, and find that these methods fail to achieve comparable performance. Our results highlight the importance of format-faithful supervision and consistency verification, and suggest that carefully designed single-model approaches can outperform more complex systems in high-stakes professional reasoning tasks. Our dataset and codes are publicly available.