JSynFlow: Japanese Synthesised Flowchart Visual Question Answering Dataset built with Large Language Models

AI summary

Overview

• Research area: Computer Vision / Vision-and-Language Models (VLM), specifically visual question answering (VQA) over flowchart images.
• Technical level: Intermediate. The paper assumes familiarity with VLMs, LLM prompting, LoRA fine-tuning, and BERTScore, though the dataset construction pipeline is described step by step.
• Scope: The paper introduces JSynFlow, a synthetic Japanese flowchart VQA dataset generated end-to-end with a large language model, and demonstrates that fine-tuning VLMs on it improves flowchart QA performance.

What This Paper Is About

VLMs are increasingly expected to read and reason about complex business documents containing flowcharts, but building the large-scale, image-and-text-aligned datasets needed to train them is extremely time-consuming. This paper addresses that bottleneck by using an LLM to synthesise an entire Japanese flowchart VQA dataset: task descriptions for many occupations, flowchart images rendered from domain-specific language (DSL) code, and matching question-answer pairs. The goal is to show that such synthesised data is sufficient to substantially improve VLM performance on flowchart-based QA.

Key Contributions

1. A synthesised Japanese flowchart VQA dataset (JSynFlow). The dataset covers 9 industries, 115 occupations, and 1,511 tasks (1,359 training, 152 test), with 11,137 QA pairs in total (10,072 training, 1,065 test), released publicly on Hugging Face.
2. A four-stage LLM-based generation pipeline. Task lists, task procedures, flowchart DSL code, and QA pairs are each generated by an LLM, with explicit review/revision loops (Algorithms 1–4) rather than single-shot generation.
3. A compilation-and-repair loop for DSL code. Generated Mermaid code is rendered with mermaid-cli; code that fails to compile is fed back to the LLM together with examples of correct syntax for correction.
4. Empirical evidence that fine-tuning on the dataset helps. Fine-tuning two VLMs with LoRA on JSynFlow raised BERTScore F1 on the held-out test set for both models.

Main Findings

• Dataset composition: JSynFlow contains 9 industries, 115 occupations, 1,359 training tasks, 152 test tasks, 1,511 tasks total, 10,072 training QA pairs, 1,065 test QA pairs, and 11,137 QA pairs in total (Table I).
• Generation settings: The LLM used was Llama-3.1-405B-Instruct, the DSL was Mermaid, and the revision iteration counts were set to N^r = N^c = 1.
• LLaVA-JP improvement: BERTScore before fine-tuning was Precision 0.6773, Recall 0.6546, F1 0.6605; after fine-tuning with JSynFlow it was 0.7627, 0.7792, and 0.7691 respectively.
• Qwen2-VL-2B-Instruct improvement: BERTScore before fine-tuning was Precision 0.8329, Recall 0.8578, F1 0.8597; after fine-tuning with JSynFlow it was 0.9372, 0.9360, and 0.9397 respectively.
• Both models benefit: Fine-tuning on the proposed dataset "yielded substantial performance improvements" for both evaluated VLMs compared with their baselines.
• Manual cleaning was necessary: Because some QA pairs were invalid due to LLM hallucinations, they were manually removed or corrected.
• Quality caveats: Dataset quality depends heavily on LLM output and currently requires manual verification and correction, and the generated images come from a standard diagram renderer, so they show less visual diversity than real-world data.

Methodology in Plain English

The construction proceeds in four stages, each driven by a prompt to Llama-3.1-405B-Instruct:

1. Task lists. The LLM first proposes 10 widely known, non-IT-skewed industry/occupation pairs in CSV form. It then expands each industry to between 2 and 20 additional occupations, and each occupation to between 2 and 20 specific advanced/complex tasks.
2. Task procedures. For each task, the LLM writes a Markdown manual written from the perspective of an experienced practitioner, including conditional branches. A junior-persona prompt then lists points of confusion, and a senior-persona prompt revises the manual to resolve them, producing a refined procedure.
3. Flowcharts. The LLM converts each procedure into Mermaid flowchart code. The code is compiled to an image with mermaid-cli; failures are re-prompted with correct syntax examples. Successfully compiled code is retained as an example for later repairs.
4. QA pairs. For each successfully compiled flowchart, the LLM generates at least 5 question-answer pairs in JSON format, with the instruction that answers must be derivable from the flowchart content alone and questions must not be about Mermaid syntax itself.

The generated flowcharts and QA pairs were then split randomly into training and evaluation sets. For the application experiment, the authors applied LoRA fine-tuning to LLaVA-JP (a LLaVA variant previously fine-tuned on Japanese data) and Qwen2-VL-2B-Instruct, using the default settings of public fine-tuning scripts except that the number of epochs was set to 3. Evaluation used the test split: for each item the model received the flowchart image plus its question, and the predicted answer was compared with the ground truth using BERTScore computed with a multilingual BERT model, averaged before and after fine-tuning.

Why This Matters

• Impact on research: The paper offers a reproducible recipe for using LLMs to manufacture diagram-centric VQA data, and it releases the resulting dataset, giving VLM researchers a ready-made resource for a document type that is underrepresented in existing benchmarks.
• Real-world applications:
  • Business process understanding, where flowcharts encode procedures that text-only explanations do not capture.
  • Knowledge sharing inside organisations, letting staff query process diagrams directly in natural language.
  • Workflow optimisation, where automated reading of process diagrams supports review and improvement.
  • Japanese-language document analysis, addressing a language setting that is often underserved in VLM training data.
• Industry relevance: The dataset is designed around occupational tasks across 9 industries, including finance, IT, medical, and legal examples listed in the paper, which maps the data onto enterprise use cases rather than academic toy diagrams.

Future Directions

• Reducing reliance on manual verification. The authors state that dataset quality currently depends on LLM output and requires manual checking and correction; more robust synthesis methods are named as future work.
• Increasing visual diversity. Generated images come from a standard diagram renderer and are less visually varied than real-world flowcharts, so improving the realism and diversity of rendered diagrams is an open problem.
• Establishing high-quality synthesis methods. The conclusion explicitly calls for "more robust methods for high-quality dataset synthesis."
• Extending beyond the current setup. The paper reports only one LLM, one DSL, and one revision setting (N^r = N^c = 1), leaving the effect of other LLMs, DSLs, or additional revision iterations unexplored in the reported content.

Target Audience

Researchers and engineers working on vision-and-language models, document understanding, or multimodal QA who need training or evaluation data for diagram-heavy documents; practitioners building Japanese-language document AI for business processes; and anyone interested in LLM-driven synthetic dataset construction pipelines.

Authors’ abstract

Vision and language models (VLMs) are expected to analyse complex documents, such as those containing flowcharts, through a question-answering (QA) interface. The ability to recognise and interpret these flowcharts is in high demand, as they provide valuable insights unavailable in text-only explanations. However, developing VLMs with precise flowchart understanding requires large-scale datasets of flowchart images and corresponding text, the creation of which is highly time-consuming. To address this challenge, we introduce JSynFlow, a synthesised visual QA dataset for Japanese flowcharts, generated using large language models (LLMs). Our dataset comprises task descriptions for various business occupations, the corresponding flowchart images rendered from domain-specific language (DSL) code, and related QA pairs. This paper details the dataset's synthesis procedure and demonstrates that fine-tuning with JSynFlow significantly improves VLM performance on flowchart-based QA tasks. Our dataset is publicly available at https://huggingface.co/datasets/jri-advtechlab/jsynflow.