PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models

AI summary

Overview

Research area: Natural Language Processing / Large Language Model evaluation, applied to physical computing education (Arduino-based microcontroller projects).

Technical level: Intermediate. The paper assumes familiarity with LLM benchmarking concepts and basic electronics terminology (breadboards, pins, wiring), though the framing is explicitly educational rather than engineering-research oriented.

Scope: The paper introduces PCEval, an automatically scored benchmark of 50 physical computing projects spanning four complexity levels, used to test 13 leading LLMs on logical circuit generation, physical breadboard circuit generation, and code generation from either circuit representation.

What This Paper Is About

Large Language Models are increasingly used for software development, but it is unclear how well they handle tasks where code must actually drive physical hardware—choosing components, wiring them on a breadboard, and keeping pin assignments consistent with the program. Existing evaluations of LLMs for electronics relied on slow, subjective manual expert review and largely ignored whether a design could physically be built on a breadboard at all. This paper builds a benchmark that removes human judgment from the loop, so that LLM performance on both the logical and the physical sides of hardware projects can be measured repeatably at scale.

Key Contributions

1. A new benchmark (PCEval): The authors present what they describe as the first comprehensive benchmark for physical computing that systematically evaluates both logical reasoning and physical implementation capability.

2. Scalable automated evaluation: The framework eliminates subjective manual assessment by decomposing each project into four generation tasks (D,C→L; D,C→P; D,L→C; D,P→C) and pairing each generated artifact with a reference artifact so the integrated system can be simulated and checked with standardized metrics.

3. Identification of a "physical gap": The evaluation surfaces a large, consistent drop between logical circuit generation and physical breadboard layout generation, with pin conflicts and breadboard bypasses identified as the dominant failure causes.

4. Educational grounding and pedagogical validation: The benchmark design is motivated by interviews with eight experienced CS educators, and after benchmarking, the authors ran a focus group with two physical computing educators (8 and 10 years of experience, 30 minutes each) and a survey of three pre-service CS teachers to assess classroom usability.

Main Findings

• Code generation outperforms circuit generation: Most models score higher on code tasks than circuit tasks. Claude 3.7 Sonnet reaches 63.4% on code overall versus 39.6% on circuit overall; GPT-4.1 reaches 64.2% versus 31.2%.

• Physical circuit generation is the hardest task: Across all models, success on D,C→P is markedly lower than on D,C→L. Many prominent models scored below 10% on physical circuit generation. Even the strongest model, o3-mini, recorded 45.2% on physical circuit generation versus 66.0% on logical circuit generation; Claude 3.7 Sonnet scored 13.6% versus 65.6%.

• o3-mini leads overall: o3-mini achieves the highest total overall success rate at 61.7%. It is followed by Claude 3.7 Sonnet (51.5%), GPT-4o (50.6%), and GPT-4.1 (47.7%). The lowest overall scores come from LLaMA 3.1 (18.3%) and DeepSeek-Coder-V2 (19.5%).

• Performance degrades sharply with project complexity: On physical circuit generation, success drops from Level 1 to Level 4 for o3-mini (65.0% → 25.5%), GPT-4o (56.7% → 12.7%), Claude 3.7 Sonnet (33.3% → 5.5%), and Mistral-Small 3 (36.7% → 3.6%). Logical circuit and code tasks also degrade with level, but far less steeply.

• Breadboard mechanics, not logic, drive physical failures: Average pin-conflict counts per sample are the highest of all error types (Claude 3.7 Sonnet: 7.52; o3-mini: 4.20; GPT-4o: 2.07). Bypass counts vary more by model (Gemini-2.0-Flash: 2.73 versus o3-mini: 0.01). Other integrity issues—extraneous, isolated, or missing components—occur in both logical and physical tasks but are smaller in magnitude and less predictive of outright failure.

• Self-improvement helps: Giving models structured failure logs and letting them refine their output over up to five turns produced consistent gains across all four tasks; o3-mini improved from 61.7% to 76.5%.

• Chain-of-thought prompting helps unevenly: Guiding models to produce an intermediate logical circuit before the final code or physical layout improved some models notably (GPT-4o: +10.4%; Mistral-Small 3: +18.0% on the physical code task) but the effect was not uniform.

• Dataset complexity gradient: The 50 projects are split into Level 1 (12 projects), Level 2 (13), Level 3 (14), and Level 4 (11). Lines of code rise from 14.00 to 27.00, cyclomatic complexity from 3.58 to 7.18, logical connections from 7.25 to 16.36, and physical connections from 15.42 to 35.0 across levels.

• Educators and pre-service teachers see value and barriers: Participants highlighted automated circuit verification as a way to reduce instructor workload and support less-experienced teachers, but flagged the lack of step-by-step assembly guidance, declining readability in complex layouts, and the absence of pedagogical conventions such as consistent wire colors. Pre-service teachers rated low-complexity projects as highly usable, with educational value dropping sharply at higher complexity.

Methodology in Plain English

The authors first interviewed eight experienced computer science educators (six in-service teachers at public middle and high schools, plus two instructors from private educational settings, roughly 30 minutes each) to identify the real pain points of teaching physical computing. Those interviews pointed to hardware setup time, difficulty giving individualized feedback, and the tangled relationship between circuit construction and code.

From there they built a dataset of 50 Arduino Uno projects organized into four complexity levels, from single-component control to multi-component interactive systems using components such as LEDs, sensors, and displays. Each project instance has five parts: a natural language description (D), a logical circuit map (L), a physical breadboard-level circuit (P), executable code (C), and an automated test procedure (T).

They then defined four generation tasks. In two of them the model receives the description plus code and must produce either the logical circuit or the physical circuit. In the other two the model receives the description plus a circuit (logical or physical) and must produce code. The trick that makes evaluation automatic is pairing: a generated circuit is combined with the reference code, and a generated code is combined with the reference circuit, so the resulting complete system can be run through the test procedure in a simulation environment. Success requires passing the simulation, and for physical circuits it additionally requires avoiding pin conflicts and breadboard bypasses that would make construction impossible. Each task-model combination was run five independent trials and results were averaged. Circuit validation also flags redundant connections, extraneous or missing components, and isolated components.

Why This Matters

Impact on research: PCEval reframes LLM hardware evaluation from "did an expert like this schematic?" to "does this artifact pass an automated test and can it be physically built?" That shift makes results reproducible and comparable across models. It also isolates a specific capability gap—physical-constraint reasoning about pins, shared nodes, and board topology—that pure code benchmarks are blind to, giving model developers a concrete target.

Real-world applications:

• Classroom assistance tools that generate and automatically verify student breadboard projects, reducing the grading and troubleshooting burden teachers reported spending over 60% of class time on.
• Automated circuit validation for instructors with limited hardware expertise, letting less-experienced teachers run physical computing courses.
• Scaffolding systems that guide students step by step rather than handing over ready-made answers, addressing automation bias and the moderating role of prior knowledge raised in the paper.
• Simulation-first tooling that lets students iterate on wiring before committing to physical components, sidestepping the setup delays and hardware breakage educators described.

Industry relevance: The benchmark connects to adjacent work on LLM-assisted Verilog/HDL generation, SPICE-based analog circuit design, and PCB layout assistance. The finding that models falter on physical layout constraints while performing acceptably on code and logic is directly relevant to anyone building AI copilots for embedded systems, IoT prototyping, or educational hardware platforms.

Future Directions

• Closing the physical-constraint gap: The paper frames physical-constraint reasoning—allocating and routing connections without violating shared nodes and board topologies—as the core weakness. How to train or prompt models to reliably handle it remains open.

• Extending beyond functional correctness: The authors argue benchmarks like PCEval should evolve to incorporate educational usability metrics such as readability conventions, step-by-step guidance, and pedagogical clarity.

• Broadening platform and scope: Current work focuses on Arduino Uno with introductory-level STEM projects. The authors report a cross-platform check using ESP32 that confirmed equivalent functional behavior without changing tasks or metrics, but professional scenarios involving power optimization, real-time requirements, and EMI considerations are excluded.

• Improving mitigation strategies: Self-improvement and chain-of-thought prompting gave only partial and uneven gains, leaving room for methods that generalize across models and complexity levels.

Target Audience

This paper is most useful to researchers building and evaluating LLM benchmarks, model developers working on hardware-adjacent reasoning and embedded-systems assistance, and computer science education researchers and curriculum designers. Physical computing instructors and teacher trainers will also find the educational usability findings and the interview themes directly relevant, as will developers building AI tools for STEM classrooms.

Authors’ abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, including software development, education, and technical assistance. Among these, software development is one of the key areas where LLMs are increasingly adopted. However, when hardware constraints are considered-for instance, in physical computing, where software must interact with and control physical hardware -their effectiveness has not been fully explored. To address this gap, we introduce \textsc{PCEval} (Physical Computing Evaluation), the first benchmark in physical computing that enables a fully automatic evaluation of the capabilities of LLM in both the logical and physical aspects of the projects, without requiring human assessment. Our evaluation framework assesses LLMs in generating circuits and producing compatible code across varying levels of project complexity. Through comprehensive testing of 13 leading models, \textsc{PCEval} provides the first reproducible and automatically validated empirical assessment of LLMs' ability to reason about fundamental hardware implementation constraints within a simulation environment. Our findings reveal that while LLMs perform well in code generation and logical circuit design, they struggle significantly with physical breadboard layout creation, particularly in managing proper pin connections and avoiding circuit errors. \textsc{PCEval} advances our understanding of AI assistance in hardware-dependent computing environments and establishes a foundation for developing more effective tools to support physical computing education.