CrochetBench: Can Vision-Language Models Move from Describing to Doing in Crochet Domain?

AI summary

Overview

Research area: Multimodal AI evaluation — specifically, whether vision-language models (VLMs) can move from describing visual content to producing executable, step-by-step procedures. The paper sits at the intersection of multimodal benchmarking, procedural reasoning, and program synthesis.

Technical level: Intermediate. Readers need basic familiarity with vision-language models, benchmark design, and the idea of a domain-specific language (DSL), but no prior crochet knowledge is required — the paper includes a crochet primer appendix.

Scope: The paper introduces CrochetBench, a 6,085-pattern benchmark built on the CrochetPARADE DSL, covering four tasks (stitch recognition, instruction selection, instruction generation, and instruction-to-DSL translation) and evaluating nine vision-language models.

What This Paper Is About

Multimodal large language models can fluently describe what they see, but it is unclear whether they can generate the step-by-step instructions needed to actually build or recreate the thing shown. Most existing benchmarks measure descriptive accuracy (captions, retrieval, alignment) rather than whether a generated procedure is correct, because verifying correctness in domains like cooking or chemistry requires slow physical execution. Crochet offers a rare workaround: patterns follow a formal symbolic grammar that can be compiled and validated automatically, so the authors build CrochetBench to test whether models can produce procedures that actually compile and preserve structural constraints.

Key Contributions

1. CrochetBench dataset. The first executable benchmark for procedural textile crafts, comprising 6,085 crochet patterns across 55 project categories, integrated with the CrochetPARADE DSL for automated validation beyond text similarity.

2. Four tasks of increasing difficulty. Stitch Recognition (Task A), Instruction Selection (Task B), Instruction Generation (Task C), and Instruction-to-DSL Translation (Task D), designed to give fine-grained diagnosis of where procedural competence breaks down.

3. Multi-model evaluation. Nine state-of-the-art vision-language models — including GPT-4o, Gemini 2.5 Flash-Lite, Claude Sonnet 4, and open-source models from 3B to 72B parameters — are benchmarked, revealing a consistent gap between recognition and executable synthesis.

4. Diagnostic analysis of failure modes. The paper shows that the gap persists across model scale and supervised finetuning, and identifies specific failure categories such as undefined stitches, unbalanced brackets, and long-range state-tracking errors.

Main Findings

• Recognition is moderate, and precision-recall trade-offs split by scale. On Task A (Stitch Recognition), Claude Sonnet 4 achieves the highest F1 at 60.94%, and DeepSeek-VL 7B leads among open models with 60.60% F1. Smaller models such as DeepSeek-VL 7B and Qwen2-VL 7B show higher recall but lower precision (DeepSeek-VL 7B recall 74.76%, precision 54.47%), while larger models such as Qwen2-VL 72B and closed-source systems favor precision over recall (Qwen2-VL 72B precision 71.86%, recall 42.68%).

• Instruction selection remains limited. Most large and closed-source models cluster around 55–60% accuracy on Task B (Gemini 2.5 Flash-Lite 55.63%, Claude Sonnet 4 57.39%, GPT-4o 58.11%), which the authors interpret as evidence that visual-textual alignment depends on shallow correlations rather than robust grounding. Qwen2-VL 72B is reported at 68.85%.

• Instruction generation collapses. Task C scores are uniformly low. Gemini 2.5 Flash-Lite is the strongest non-finetuned system at 4.93% BLEU and 30.50% ChrF; GPT-4o reaches 3.38% BLEU and 24.07% ChrF; BLIP-2 Flan-T5 XL drops to 0.22% BLEU.

• Fluent output does not imply correct procedure. In the case study, the ground truth is a seven-point star with alternating blue and brown yarn and tassels. GPT-4o and Claude produce well-formed, human-like instructions that capture local colors but misconstruct global geometry — GPT-4o produces a four-point star and begins with brown instead of blue, Claude generates an eight-point motif. Gemini is the only model to explicitly recognize the motif as a seven-point star, yet its instructions still fail to produce the correct topology. Qwen2-VL-7B and DeepSeek-VL collapse into non-star geometries, with DeepSeek-VL degenerating into a single-loop pattern.

• Early steps are the hardest in step-level DSL translation. On Task D step-level, most models achieve under 15% validity in the first two steps, improve modestly for steps 3–4, and reach only 55–65% in later steps. Errors made early propagate irreversibly.

• Scaling does not help grammar-sensitive tasks. Qwen2-VL-72B underperforms Qwen2-VL-7B, and Gemma-3-27B underperforms Gemma-3-4B on step-level translation. On project-level translation, undefined-stitch errors are far higher for larger models (Qwen2-VL-72B 72.0% vs. Qwen2-VL-7B 18.9%; Gemma-3-27B 42.4% vs. Gemma-3-4B 20.3%), suggesting a tendency toward uncontrolled symbolic invention.

• Executable programs are rare. Even the strongest models (Claude, Gemini, GPT-4o) produce only 5–8% executable programs at the project level. Smaller open-source models such as Qwen2-VL-7B and DeepSeek-VL achieve higher valid rates, but qualitative inspection attributes this largely to copying or closely following few-shot exemplars.

• Compilable does not mean correct. DINO similarity between ground-truth product images and renderings of executable programs is uniformly low across all models (0.14–0.22), far below the approximate 0.6 threshold for visually matched crochet images.

• Finetuning improves fluency, not correctness. Finetuning Qwen2-VL-7B-Instruct (AdamW, 3 epochs, effective batch size 8) raises BLEU from 1.67% to 5.64% (+238%), ROUGE-L from 21.10% to 25.10% (+19%), and ChrF from 15.99% to 22.39% (+40%), outperforming all evaluated closed-source systems. Absolute performance stays low, and outputs still contain stitch-count errors, malformed repeats, and incorrect global structure.

Methodology in Plain English

The authors collected publicly available crochet patterns from the Yarnspirations website, where patterns are distributed as PDFs. They used a GPT-4o-mini-based pipeline to parse and normalize these PDFs into structured JSON records containing metadata, materials, measurements, gauge, abbreviations, and step-by-step instructions. Each pattern was paired with a high-resolution finished-product image (typically 2000×2000 pixels, sRGB). To check parsing reliability, they sampled 494 patterns across 55 categories and found 97.1% word-level overlap with the original PDFs and 95–100% overlap within the procedural instruction sections. From each JSON record they derived four supervised datasets.

They then designed four tasks. Task A asks models to identify which stitch types appear in a finished-product image — a multi-label problem scored with precision, recall, and F1 across 6,009 examples. Task B is a four-way multiple-choice question with 6,003 examples, where one ground-truth instruction competes against three distractors drawn from the same project category, scored by accuracy. Task C asks models to generate natural-language instructions from an image across 6,009 examples, scored with BLEU, ROUGE-L, and ChrF. Task D asks models to translate natural-language instructions into CrochetPARADE DSL programs, with a 119-item step-level variant and a 6,009-item project-level variant.

For Task D, there is no canonical gold program, since many different programs can produce the same final artifact. Instead, correctness is measured by whether the generated DSL compiles and executes, reported as Valid Pattern Rate. For project-level outputs that compile, the executable portion is rendered into a crochet-like image and compared to the ground-truth product image using DINO similarity. The authors also identify the first point of failure for each invalid prediction and categorize it with an error taxonomy. All models were evaluated on a stratified 80/5/15 train/validation/test split built at the pattern level to avoid cross-task leakage, with the 119 step-level items treated as a standalone diagnostic set.

Why This Matters

Impact on research. CrochetBench offers a way to evaluate procedural competence without physical execution, which the paper argues is rare. It connects multimodal benchmarking to program-synthesis evaluation, where the test is whether output actually works rather than whether it looks similar to a reference string. The results challenge a common assumption: that larger models and more finetuning data will close gaps in structured reasoning.

Real-world applications:

• Robotics and assembly instructions. Generating step-by-step build or assembly procedures from a photo of a finished object is the same describing-to-doing gap, and the failure patterns here (misread global geometry, early errors propagating) likely transfer.

• CAD and design tools. Turning a rendered design image into an editable procedural specification is structurally similar to image-to-DSL translation.

• Procedural content generation. Games, simulations, and creative tools that need machine-generated build recipes or crafting steps.

• Craft and hobby platforms. Assisting users in converting photos of finished items into workable patterns, or checking whether a submitted pattern is internally consistent.

Industry relevance. The benchmark provides a concrete baseline for anyone building systems that must emit executable specifications rather than plausible-sounding text. Because the evaluation includes an executable validator and image-based similarity check, it offers a template for functional evaluation in domains where exact-match scoring is meaningless.

Future Directions

• Training paradigms beyond supervised finetuning. The paper explicitly notes that finetuning was limited to supervised learning on a single architecture and identifies program-guided learning and learning with execution feedback as important untested directions.

• Preventing symbolic invention at scale. Larger models produced more undefined-stitch errors, so controlling the tendency to invent stitches not defined in the DSL is an open problem.

• Generalization beyond crochet. The authors caution that the benchmark covers a single creative domain, and whether the findings transfer to robotics, mechanical assembly, or scientific experimentation remains untested.

• Improving semantic equivalence evaluation. The DSL abstracts real-world crochet, and DINO similarity is described as a coarse proxy for semantic fidelity (0.14–0.22 across all models, versus an approximate 0.6 "good" threshold), leaving room for better equivalence measures.

• Controlling for pretraining overlap. The authors state they could not fully rule out overlap between the benchmark and model pretraining corpora.

Target Audience

Researchers working on multimodal evaluation, procedural reasoning, and program synthesis will get the most from this paper, particularly those looking for benchmarks that validate outputs functionally rather than through text similarity. It is also relevant to practitioners building systems that convert visual input into executable specifications, and to anyone studying the limits of model scaling for structured, grammar-sensitive tasks. Readers without a machine learning background can follow the case study and task descriptions, though the evaluation metrics and DSL details are aimed at a technical audience.

Authors’ abstract

While multimodal large language models can describe visual content, their ability to generate executable procedures remains underexplored. CrochetBench presented in this paper evaluates this shift from describing to doing through fine-grained procedural reasoning in crochet: models must recognize stitches, select structurally appropriate instructions, and generate compilable procedures. We adopt the CrochetPARADE DSL as our intermediate representation, enabling structural validation and functional evaluation via execution. The benchmark covers tasks including stitch classification, instruction grounding, and both natural language and image-to-DSL translation. Across all tasks, performance sharply decreases as the evaluation shifts from surface-level similarity to executable correctness, revealing limitations in long-range symbolic reasoning and 3D-aware procedural synthesis. Our proposed CrochetBench offers a new lens for assessing procedural competence in multimodal models and highlights the gap between surface-level understanding and executable precision in real-world creative domains. Code is available at https://github.com/Peiyu-Georgia-Li/crochetBench.