PCFBench: How Far Can Large Vision-Language Models Go in Physics-aware Photonic Inverse Design?

AI summary

Overview

Research area: Multimodal AI benchmarking at the intersection of computer vision, vision-language models (VLMs), and photonic engineering (specifically photonic crystal fiber design).

Technical level: Intermediate. Readers should be comfortable with the general idea of VLMs and multimodal evaluation, but no specialist electromagnetics background is required to follow the argument.

Scope: The paper introduces PCFBench, the first benchmark that tests whether vision-language models can move from seeing a photonic crystal fiber cross-section to making physically grounded inverse-design decisions about it.

What This Paper Is About

Photonic crystal fibers are designed today through a slow, manual loop: an engineer looks at a cross-section micrograph and a mode-field image, infers optical properties, diagnoses what is wrong, proposes a geometric change, and re-runs a finite-difference time-domain (FDTD) simulation to check. The paper asks whether general-purpose vision-language models can take over any part of that loop, or whether they only recognize visual patterns without reasoning about the underlying waveguide physics. To answer this, the authors build PCFBench: 26 tasks across six cognitive levels, drawn from over 220K FDTD simulation samples spanning 18 fiber families, evaluated on both individual models and multi-agent pipelines.

Key Contributions

1. A unified benchmark for photonic design. PCFBench defines a six-dimensional evaluation hierarchy (Geometry Perception, Physics Understanding, Text Generation, Multi-modal Reasoning, Inverse-Design Decision, Code Generation) spanning 26 tasks, with a dual evaluation paradigm covering both isolated single-model tasks and sequential multi-agent pipelines.

2. A large-scale, expert-annotated, physics-grounded dataset. The corpus contains over 220K FDTD simulation samples across 18 structurally diverse fiber families, each paired with structure and mode-field images at two telecom wavelengths (1310 and 1550 nm) and enriched with four levels of expert annotation: L1 templated geometry descriptors, L2 rule-grounded physics QA, L3 expert-authored mechanism hypotheses with design suggestions, and L4 executable solver scripts.

3. New empirical insights into VLM limitations. The authors evaluate 41 frontier VLMs and 13 multi-agent systems, identifying a persistent perception–design gap that holds across model scales and coordination strategies.

4. A closed-loop validation protocol. Rather than scoring design proposals only against rules, the paper re-simulates proposed parameter changes in the same Meep FDTD solver to measure whether the design actually improved.

Main Findings

• Perception is the easy part; design is not. Gemini-3-Flash performs strongly on geometry perception (family accuracy .668, counting accuracy .757), yet no model reaches 50% on topology recognition. Perception correlates weakly with inverse-design success.

• The perception–design gap is explicit in individual models. Qwen3.5-122B-A10B leads on topology accuracy (.495) but achieves only .355 closed-loop success, while the much smaller Qwen2.5-VL-7B reaches .638.

• Bigger is not always better. Within the Qwen-VL family, scaling from 7B to 235B improves topology recognition by 25 percentage points but degrades closed-loop success by 47 percentage points. GPT-4o-mini outperforms both GPT-4o and GPT-5 on closed-loop design, and Llama-4-Maverick (400B) ranks below several smaller models in the aggregated global ranking.

• Scaling curves diverge by category. Geometry perception improves with parameter count (R² > 0.4), whereas inverse-design decisions remain flat or decline.

• In-context examples help perception, not design. Eight in-context exemplars raise family accuracy by up to .161, while closed-loop success moves by at most .009.

• The gap is physical, not visual. Families farther from natural images in SigLIP space are recognized less accurately (Spearman ρ ≈ −.6), but that distance is unrelated to closed-loop success. GPT-4o-mini keeps the highest closed-loop success of the benchmark (about .80) while recognizing fewer than a third of the families.

• Text generation and code syntax saturate. BERTScore-F1 for geometry description clusters in a narrow 0.27–0.36 band across models, and simulation-code syntax plus parameter-modification completeness are near-saturated — models write valid Meep scripts even when they cannot reason about the encoded physics.

• Paired prompts expose majority-class shortcuts. Several models score perfect macro-F1 on Field-only mechanism diagnosis by defaulting to the majority class while scoring near zero on the Paired variant with all seven options.

• The best multi-agent strategy depends on the backbone. MAD leads on GPT-4o-mini (closed-loop .731) but falls to .411 on Qwen-VL-Max, where a single Vanilla call reaches .580. EvoMAC, worst on GPT-4o-mini (.232), becomes best on Qwen-VL-Max (.732).

• More agents does not mean better design. On GPT-4o-mini, two-agent AutoGen (closed-loop .680) beats DyLAN, MacNet, MAV, and EvoMAC, all of which use more agents.

• Domain decomposition helps but cascades errors. The Sequential Pipeline (Cascaded) consistently surpasses generic multi-agent baselines on parameter selection (e.g., .655 on GPT-4o-mini versus .731 best generic result but higher parameter-selection accuracy than most), and its Oracle mode shows substantial headroom (.785 versus .655 closed-loop on GPT-4o-mini). Upstream perception errors propagate downstream at a cascade rate exceeding 60%.

• Rankings transfer to real fabricated fibers. On 120 SEM micrographs covering 10 families, six models keep exactly their Family Classification order from a matched simulated subset (Kendall τ = 1), though every model loses accuracy (Gemini-3-Flash drops from .642 to .525).

• Propagation-level reasoning is harder still. On 600 index-guiding fibers, no model exceeds .69 on dispersion-regime classification or .59 on pulse-broadening regime, with errors clustering near the regime boundary.

• Closed-loop design succeeds mainly on single-axis monotonic edits. On an anti-resonant fiber, 9 of 11 systems improved loss (six by 84 to 97 dB, three to numerically zero loss), mostly by retuning tube wall thickness. On a dual-core coupler, 10 of 11 improved, most by adding cladding rings. Joint two-parameter edits change outcomes by at most one case out of 20, whereas three feedback rounds add two to five improved cases.

• Code generation is decoupled from photonic reasoning. The cross-task Spearman correlation matrix shows code generation largely independent of the other categories, and geometry tasks cluster tightly among themselves while correlating weakly with inverse-design decisions.

Methodology in Plain English

The authors built the benchmark in six stages. First, they assembled 18 structurally diverse fiber families — conventional telecom designs, hollow-core anti-resonant fibers, and specialty geometries such as plasmonic, vortex ring-core, and fractal quasicrystal structures. Second, they ran one unified Meep FDTD solver configuration at 1310 and 1550 nm across all samples so results stay comparable between families. Third, they aggregated geometric parameters (pitch, hole diameter, strut thickness, core size, ring count) with the simulated physics into a 220K-sample corpus, with structure images rendered at 336×336 resolution.

Fourth, photonics experts layered four annotation levels onto every sample, from templated geometry descriptions up to expert-written mechanism hypotheses and executable solver scripts. Fifth, annotations were re-checked, assessed for structural and textual rationality, and verified by re-simulation, with inconsistent samples removed.

From this corpus they defined 26 tasks in a hierarchy of increasing difficulty and evaluated models two ways: single-model, where each VLM is queried independently with a fixed temperature of 0.1; and multi-agent, where the design workflow is decomposed into a four-stage Sequential Pipeline (Perception → Understanding → Reasoning → Design) and compared against 12 generic coordination strategies in the MASLab framework across three backbones (GPT-4o-mini, Qwen-VL-Max, Gemini-3.1-Lite). Ground truth for design-direction and closed-loop tasks comes from family-specific expert rules that account for non-monotonic regimes such as anti-resonant strut bands, and a composite Human Score from two photonics experts across six dimensions complements the automated metrics.

Why This Matters

The paper shifts the question for scientific AI benchmarks from "can the model describe what it sees" to "can the model act correctly on what it sees." It shows that strong geometric perception, larger parameter counts, and elaborate multi-agent scaffolding all fail to produce physically sound design decisions — an argument that bears on how the community builds and evaluates scientific agents generally, not just in photonics.

Real-world applications of the underlying technology, as framed in the paper:

• Telecommunications: ultra-low-loss and low-latency fiber transmission.
• High-power laser delivery: anti-resonant and nested anti-resonant fibers for short-pulse delivery in the UV to mid-infrared.
• Biochemical and gas sensing: suspended cobweb and hollow-core fibers with high evanescent-field fraction, plus gas-cell sensing where low nonlinearity matters.
• Quantum-state preservation and supercontinuum generation: specialty geometries whose modal properties must be tightly controlled.

Industry relevance: PCF design today is manual and expert-intensive, with each iteration costing hours of FDTD simulation plus trial-and-error tuning. A reliable VLM-based design assistant would compress that loop. PCFBench gives industry a shared yardstick for measuring progress, and its finding that simulation-grounded supervision can outweigh raw scale is directly actionable for teams deciding whether to buy bigger models or build domain-specific ones.

Future Directions

• Simulation-grounded supervision. The authors argue that structure–field–loss data from FDTD simulation provides signals absent from internet-scale pretraining, and that domain-specific adaptation may matter more than scale.

• Rewarding feasibility and bounded updates. Closed-loop success depends on magnitude, constraints, and multi-parameter coordination, not just direction, so training signals should reward consistency and bounded parameter changes rather than one-shot direction prediction.

• Solver feedback inside the loop. Integrating simulation feedback — either as tool use at inference time or as a reward for closed-loop tuning — is proposed as a higher-yield direction than further model scaling.

• Open questions the benchmark surfaces: whether explicit physical grounding can be taught rather than prompted; how to prevent error cascades above 60% in sequential agent pipelines; and whether the tractability of single-axis monotonic edits can be extended to nested and quasi-crystalline geometries that require coordinated multi-parameter changes.

Target Audience

Researchers building or evaluating scientific agents and multimodal benchmarks will find the most value here, particularly those working on physics-aware reasoning, tool-augmented design loops, or multi-agent error propagation. Photonics and fiber-optics engineers interested in automating inverse design will benefit from the task taxonomy and the negative results about off-the-shelf VLMs. Machine-learning practitioners deciding between model scaling and domain adaptation for specialized engineering tasks will find the scaling analysis directly relevant.

Authors’ abstract

Photonic crystal fibers (PCFs) underpin critical applications in telecommunications, sensing, and high-power laser delivery, yet their design remains a manual, expert-intensive process that couples geometric intuition with deep waveguide physics. Vision-language models (VLMs) offer a promising path toward automating PCF design, but can they truly reason about photonic physics from microscopy images, or do they merely recognize visual patterns? We introduce PCFBench, the first comprehensive benchmark for evaluating VLMs on the perception, reasoning, and decision steps of the photonic inverse-design loop. PCFBench spans 26 tasks organized into six categories of increasing cognitive demand (Geometry Perception, Physics Understanding, Text Generation, Multi-modal Reasoning, Inverse-Design Decision, and Code Generation), built from over 220K FDTD simulation samples across 18 structurally diverse fiber families. We evaluate both individual VLMs on isolated tasks and multi-agent systems (MAS) on the full sequential pipeline from perception to design. Our results reveal a striking perception-design gap: current models can identify fiber geometry with moderate success, yet consistently fail to translate that visual understanding into physics-aware design actions. Moreover, cascading agents through the pipeline exposes significant error propagation, where upstream perception mistakes compound into downstream design failures. These findings suggest that neither model scaling nor naive agent chaining suffices, and that closing the loop demands explicit physical grounding. PCFBench provides the community with a rigorous, multi-dimensional testbed to measure progress toward AI-assisted photonic engineering. The project is available at https://pcfbench.github.io/.