Crucible: Quantifying the Potential of Control Algorithms through LLM Agents

AI summary

Overview

Research area: Artificial intelligence and control systems — specifically the use of large language model (LLM) agents to evaluate and improve control algorithms, spanning classic control, adaptive bitrate (ABR) video streaming, and distributed task scheduling.

Technical level: Advanced. The paper combines control theory, LLM-agent design, Bayesian optimization, and a formal metric for comparing task environments, and it assumes familiarity with terms like QoE, DAG scheduling, PID/LQR, and hyperparameter optimization.

Scope: The paper introduces Crucible, an LLM-driven "multi-level expert simulation" agent plus a formalized metric, and uses them to quantify the "Tuning Potential" of control algorithms across simulation benchmarks and a real-world Dash.js deployment.

What This Paper Is About

Control algorithms are usually judged by how they perform out of the box, with default parameters, in ideal conditions. But in practice, domain experts tune them — sometimes only by adjusting numbers, sometimes by rewriting the control logic itself — and the paper argues that how much a given algorithm can be improved this way is an overlooked but important property.

The paper defines this property as Tuning Potential and builds Crucible to measure it. Crucible simulates developers of different skill levels with an LLM agent, then uses a formalized performance-profile-based similarity metric to normalize and score how much each algorithm gains from being tuned across a range of task environments.

Key Contributions

1. Formalizes Tuning Potential as a distinct and previously neglected dimension of algorithm evaluation, arguing that ignoring it limits the real-world impact of algorithms and keeps it from becoming an explicit design objective.

2. Proposes Crucible, described as the first system designed to quantify the Tuning Potential of control algorithms, combining an LLM-based multi-level expert simulation agent with a formalized environmental metric that enables comparison across different domains.

3. Defines a normalized potential metric built on "performance characteristic vectors" derived from a set of probe algorithms, using root mean square error to measure environment distance and a similarity-weighted average performance gain as the potential score.

4. Validates the framework broadly across classic control (Cart-Pole), computer systems (ABR and scheduling), a real-world Dash.js deployment, multiple LLMs, and robustness tests, then shows the resulting insights can guide algorithm redesign.

Main Findings

• Logic-level tuning beats parameter tuning alone. On Cart-Pole, the simple Bang-bang controller scored 34 initially, 56 after one Bayesian optimization step, and 500 after one LLM-driven logic modification — matching the DQN reference score of 500. PID went from 34 to 77 (one Bayes) to 110 (one LLM) to 271 (two Bayes) to 500 (two LLM), while LQR reached 500 after a single Bayesian step.

• Crucible expands the optimization space in complex domains. Across the computer-system benchmarks, Crucible's logic-level improvements yielded gains of up to 44.1% on the Puffer dataset relative to Bayesian optimization results.

• LLM logic changes need Bayesian optimization to be reliable. Without Bayesian fine-tuning, approximately 60% of test scenarios showed no significant performance gains, which the authors attribute to LLM limitations in fine-grained numerical operations. Adding Bayesian optimization cut the proportion of ineffective test scenarios from 60% to 20%.

• Tuning can beat more complex designs in ABR. In the motivation experiments, HYB optimized via Bayesian optimization (HYB(B)) and HYB adjusted via Crucible (HYB(C)) reduced video playback stalling time by 92%. Parameter-optimized HYB(B) still could not surpass BOLA and RobustMPC on QoE, but logic-adjusted HYB(C) achieved the best QoE, raising the ranking from lowest to fourth and first place respectively.

• Simple scheduling logic can win after tuning. The simple FIFO algorithm tuned by Crucible achieved sub-optimal executor utilization and the optimal cumulative waiting time in the scheduling problem.

• Real-world ABR gains transfer. In a Dash.js deployment over public WiFi, Crucible-tuned HYB and BBA both reached a QoE score of 1.72, up from 1.40 and 1.56 respectively and above the RL-based Pensieve at 1.66. BOLA rose from 1.20 to 1.54 and MPC from 1.72 to 1.79, while Pitree showed no improvement (1.73 to 1.73).

• Results hold across LLMs. With 3 LLM iterations and 20 Bayesian iterations, Claude 3.5 Sonnet, GPT-4o-mini, and Claude 3.7 Sonnet all improved the ABR algorithms and preserved the relative ranking. Claude 3.7 Sonnet scored highest on HYB (1.12 versus 0.92 initial), while for Pitree all models stayed low (0.35, 0.37, 0.36 versus 0.31 initial).

• Two factors govern potential. The authors identify an algorithm's representational capacity and its comprehensibility as the primary influences on tuning potential. HYB, with a broader state space, showed greater potential (0.068 ± 0.117, ideal environment FCC) than single-state BBA (0.018 ± 0.008, ideal environment Oboe), and the decision-tree-based Pitree had low potential (0.033 ± 0.022, ideal environment Puffer) despite its limited initial performance.

• More optimization capacity helps, with diminishing differences by domain. Increasing Bayesian iterations from one to two produced a notable leap: in ABR the improvement rate with zero Bayesian iterations rose from 9.54% to 29.91%, while in scheduling the proportion of scenarios failing to improve decreased from 80% to 60%. Improvement magnitudes were lower for scheduling than for ABR because scheduling inputs (DAG graph information and node states) are more complex than ABR's buffer size and bandwidth.

• Insights translate into redesign gains. Enhancing BBA's representational capacity produced BBA_C, which started within 0.5% of original BBA but consistently outperformed it across optimization iterations, ending with a 4% performance improvement. Conversely, simplifying logic paid off in scheduling: after Crucible tuning, SJF achieved shorter task completion times than the initially stronger Multilevel Feedback algorithm.

Methodology in Plain English

Crucible treats algorithm tuning as a simulation of what human developers do, then scores the results in a comparable way.

The agent first receives domain knowledge through system prompts covering the task description, the optimization objectives, and the testing environments. It can also call standardized tools such as Bayesian optimization to tune numerical parameters. Crucible supports two distinct dimensions of improvement: expanding an algorithm's control space (representational capacity) through automated optimization, and restructuring its logic (dependent on comprehensibility) through LLM reasoning.

Each modification is stored as a triplet of rationale, action, and observed result, and this history is fed back to the LLM before the next round so it can learn from prior attempts. Developer skill levels are simulated not by changing prompts but by changing the computational budget: the number of Bayesian optimization calls (0, 10, or 20 iterations) and the number of reflection iterations (1, 2, or 3).

The systems being evaluated expose a standard interface: the algorithm's code plus execution logs made of state–action–result triplets. The LLM may only modify the control algorithm, and it obtains new results by invoking the original execution file.

For scoring, the framework first runs a small set of probe algorithms across all test environments and records their scores. Each environment's scores are normalized across the environment set into a performance characteristic vector, which acts as a quantitative "fingerprint." The distance between two environments is the root mean square error between their vectors, and similarity is defined as the maximum of 0 and 1 minus that distance. An algorithm's potential is then the average performance gain from original to Crucible-tuned performance across all test environments, weighted by the similarity of each test environment to the algorithm's ideal environment. Gains in environments very dissimilar to the ideal one are down-weighted.

Why This Matters

Impact on research. The paper reframes algorithm evaluation: instead of comparing only default or ideal-condition performance, it asks how much headroom an algorithm offers a tuner. The authors argue this should be treated as a first-class design goal and even as an optimization metric, not a post-deployment afterthought.

Real-world applications.

• Video streaming: ABR algorithms such as BBA, MPC, HYB, BOLA, and Pitree determine bitrate selection and directly affect stalling and quality of experience for viewers.
• Distributed data processing: DAG task schedulers such as SJF, FIFO, Multilevel Feedback, and Tetris control how Spark cluster tasks are allocated and how long jobs take.
• Robotics and autonomous vehicles: The paper cites gait control for robots and motion control for autonomous vehicles as control domains where expert tuning is routine.
• Industrial automation and data centers: The introduction lists industrial automation and data center scheduling as settings where control algorithms dynamically regulate system behavior.

Industry relevance. Production environments expose stable, predictable scenario features — user bandwidth patterns in content provider services and system load patterns in scheduling — that allow targeted optimization beyond general-purpose defaults. Crucible offers a way to predict, before deployment, which algorithms are worth the cost of expert tuning, and the real-world Dash.js results demonstrate that the simulated gains carry over to live conditions.

Future Directions

• Understanding and adjusting black-box algorithms. The authors state that they currently cannot directly modify the internal logic of black-box algorithms, which is why they analyzed decision trees distilled from black-box algorithms; tackling black-box internals remains open.

• Stabilizing LLM-dependent results. Because the framework is built on an LLM, different capabilities and versions of LLMs may influence outcomes. The authors argue this still has value since different LLMs approximate developers with different skill levels, but the stability problem is unresolved.

• Establishing potential as a design objective. The paper calls for treating Tuning Potential as something evaluated from the outset, which raises the question of how algorithm designers should build representational capacity and comprehensibility into new algorithms deliberately.

• Designing for comprehensibility. The Pitree results and the reference to ComTree suggest a research direction of designing algorithms whose logic LLMs and developers can more easily interpret and restructure, particularly in high-dimensional domains like DAG scheduling.

Target Audience

This paper is most useful to researchers and practitioners who design, evaluate, or deploy control algorithms — including network and streaming engineers working on ABR, distributed-systems engineers working on schedulers, and control theorists — as well as researchers applying LLM agents to software engineering, automated optimization, or simulated expert behavior. It also suits algorithm designers interested in why some algorithms adapt well to tuning and others do not, and readers tracking new evaluation metrics for machine-learned and heuristic control systems.

Authors’ abstract

Control algorithms in production environments typically require domain experts to tune their parameters and logic for specific scenarios. However, existing research predominantly focuses on algorithmic performance under ideal or default configurations, overlooking the critical aspect of Tuning Potential. To bridge this gap, we introduce Crucible, an agent that employs an LLM-driven, multi-level expert simulation to turn algorithms and defines a formalized metric to quantitatively evaluate their Tuning Potential. We demonstrate Crucible's effectiveness across a wide spectrum of case studies, from classic control tasks to complex computer systems, and validate its findings in a real-world deployment. Our experimental results reveal that Crucible systematically quantifies the tunable space across different algorithms. Furthermore, Crucible provides a new dimension for algorithm analysis and design, which ultimately leads to performance improvements. Our code is available at https://github.com/thu-media/Crucible.