Where Natural Language Ends in Agents: Executable Skill Composition with MetaSkill

AI summary

Overview

• Research area: AI agents, tool/skill composition, workflow orchestration, and hybrid language–symbolic representations for LLM-based systems.
• Technical level: Advanced (formal graph notation, runtime semantics, blinded human-style evaluation protocol).
• Scope (one sentence): The paper proposes MetaSkill, a hybrid plan representation that makes cross-Skill dependencies and upstream-text handoffs executable in a DAG runtime while leaving each Skill's internal task judgment in natural language, and evaluates it on 20 multi-Skill AgentSkillOS tasks with two LLM backbones.

What This Paper Is About

Agent "Skills" — reusable packages of instructions, scripts, templates, and resources — are deliberately open-ended, but when several of them must work together, something has to keep track of what runs first and which upstream output gets fed into which downstream step. A plain natural-language standard operating procedure (NL-SOP) describes these constraints readably but leaves it to the model to actually honor them, so steps can start too early or grab the wrong upstream text. The paper's goal is to find the right boundary between what should stay in natural language and what should be made executable, and to test whether that boundary improves the artifacts agents produce.

Key Contributions

1. A principle of selective formalization at Skill boundaries. The authors give three explicit criteria for deciding whether a coordination relation should become executable: it must be persistent (survive across separately invoked Skills), referentially stable (name a specific node, predecessor, or stored output rather than an open-ended preference), and machine-observable (a runtime can check compliance without judging artifact quality).
2. The MetaSkill representation. A hybrid plan P_i = (Γ_i, L_i), where Γ_i = (V_i, E_i, κ_i, b_i) holds an acyclic dependency graph, Skill-package selection κ_i, and predecessor-text bindings b_i, while L_i keeps each invocation's local natural-language instruction. Complete Skill packages are preserved rather than rewritten into a workflow language.
3. Runtime semantics with auditable coordination. The runtime validates the graph (rejecting duplicate node IDs, undefined dependencies, and cycles), computes a ready set, renders predecessor text into downstream arguments, dispatches at most K ready nodes, and stores each node's text output by node identifier. A formal trace-conformance check verifies that every declared dependency's finish time precedes its dependent's start time, and that every declared predecessor-text reference resolves to a non-empty rendered value.
4. A controlled empirical study plus external comparisons. Three coordination interfaces (NL-SOP, Prompt Graph, MetaSkill) are derived as deterministic projections of one shared frozen plan per task, compared alongside a matched-Skills AgentSkillOS comparison and agent-stack comparisons with OpenSquilla, OpenClaw, and Hermes Agent.

Main Findings

• All reported preference point estimates favor MetaSkill, but most intervals include parity. Against NL-SOP under GLM-5.2, preference is 0.567 (95% task-bootstrap CI [0.433, 0.692]); against Prompt Graph under GLM-5.2, 0.550 ([0.425, 0.667]).
• Two intervals exclude parity. MetaSkill versus Prompt Graph under DeepSeek-V4-Flash reaches 0.658 with CI [0.575, 0.742] (W/T/L 28/23/9), and MetaSkill versus OpenClaw under GLM-5.2 reaches 0.625 with CI [0.525, 0.725] (W/T/L 27/21/12).
• DeepSeek-V4-Flash versus NL-SOP yields 0.558 ([0.467, 0.650], W/T/L 22/23/15).
• Matched-Skills comparison with AgentSkillOS yields 0.542 ([0.425, 0.650], W/T/L 26/13/21) when both systems receive the identical Skill packages; Skill retrieval is explicitly outside this comparison.
• Agent-stack comparisons under GLM-5.2: OpenSquilla running the NL-SOP input gives 0.567 ([0.433, 0.692]) — the same 60 artifact pairs as the GLM-5.2 NL-SOP row; Hermes Agent gives 0.575 ([0.433, 0.717]), an interval that includes parity.
• Execution audit confirms declared coordination. In GLM-5.2 traces, 264/264 planned nodes complete, there are 0/282 declared dependency/precedence violations, and 270/270 declared predecessor-text references identify a recorded node with a non-empty rendered input value.
• Parallelism occurs in practice. 18/60 audited traces show overlapping execution of ready nodes, indicating the runtime actually exploits independent branches rather than serializing everything.

Methodology in Plain English

The authors take 20 tasks from the AgentSkillOS benchmark — specifically the tasks whose released definitions designate two to four Skills, excluding the 10 one-Skill tasks because the hypothesis concerns cross-Skill relations. The 20 tasks span five categories and produce DOCX, PPTX, HTML, JSON, image, and video artifacts.

For each task they hand-build a single plan from the task specification and its designated Skills, verify its structure, and freeze it before evaluation, recording provenance, hashes, and rendering paths in a manifest. All three conditions are then generated mechanically from that same frozen plan:

• NL-SOP renders node identity, Skill, objective, prerequisites, and expected result as readable prose, and deliberately omits the fields naming which upstream text goes into which downstream argument.
• Prompt Graph exposes the full graph records including named input references, but the host agent still has to track state and resolve those references itself.
• MetaSkill exposes the same graph content but hands execution to the runtime, which waits for prerequisites and inserts the predecessor text into the invocation arguments.

Every condition runs in the same host harness with the same backbone model and the same complete Skill packages. Each task and interface gets 3 executions per backbone on GLM-5.2 and DeepSeek-V4-Flash, giving 180 runs and 360 judge calls per backbone (two display orders per artifact pair).

Artifacts are judged blind: they get opaque aliases, and a fixed anthropic/claude-opus-4.5 judge at temperature zero scores both display orders, with conflicting valid winners merged into a tie. Text and JSON files are passed directly up to 5000 characters per file; visual artifacts are rendered as images, with at most 30 images per side and videos represented as uniformly sampled frames. Preference is averaged over N = 20 tasks and R = 3 repetitions, with 95% intervals from 10,000 task-level bootstrap resamples that keep all repetitions of a task together as a block. In the experimental MetaSkills, each inserted predecessor text is capped at 6000 characters, and generated files stay in a shared workspace rather than being passed through typed ports.

Why This Matters

The paper isolates a design question that most agent systems answer implicitly: which parts of a multi-step procedure should the model interpret, and which should a runtime enforce? By holding the source plan, the Skill packages, the backbone, and the harness constant and varying only the representation and executor, it separates "can the model write a good plan" from "can the model faithfully follow one."

Real-world applications:

• Business and financial reporting pipelines where a chart-generation step must consume the exact output of a specific analysis step, not a plausible-looking neighbor.
• Multi-artifact content production (the paper's own example is an animation plus a handout of at least 8 pages, followed by a validation step that reads both) where separate deliverables must stay consistent.
• Regulated or auditable agent workflows in which an operator needs a machine-checkable record that declared ordering and handoffs actually occurred, not just a transcript.
• Reusable internal skill libraries, since a Skill package can appear in many different compositions with different objectives and bindings without any change to its internals.

Industry relevance: the results suggest that for open-ended, artifact-rich Skills, a thin coordination layer is enough to make dependencies enforceable, and that the costs of that layer are modest — no point estimate falls below parity, and two comparisons exclude it. The code is released at github.com/TokenRhythm/opensquilla.

Future Directions

• Skill retrieval at ecosystem scale. The study assumes the relevant Skills have already been identified; the authors state that retrieval from large or evolving Skill libraries is not systematically evaluated and is left for future work, including how retrieval quality interacts with MetaSkill construction and execution.
• Plan acquisition. A MetaSkill can be authored directly, adapted from an SOP, or proposed by an upstream plan generator, but only execution of a supplied, frozen plan is evaluated here — so how good generated graphs are remains open.
• Judge and evidence robustness. Artifact quality rests on a single fixed judge with text and rendered visual evidence under input limits (5000 characters per file, at most 30 images per side), so preferences may depend on that judge and on what it can see.
• Stronger statistical separation. Since most 95% intervals include parity, larger task sets or more repetitions would be needed to establish which interface differences are reliable rather than directional.

Target Audience

Agent-system builders and applied researchers who design multi-Skill or multi-tool pipelines and need reliable ordering and data handoffs; workflow-language and orchestration researchers interested in where formal structure should sit relative to natural-language instructions; and evaluation-minded readers who want a template for controlled comparisons that hold the source plan and toolset fixed while varying only the representation. The formal notation and bootstrap protocol make it more suited to readers with some background in LLM agent systems than to beginners.

Authors’ abstract

Agent Skills express open-ended task knowledge in natural language, but composing them requires persistent execution order and upstream-output handoffs. We propose selective formalization at Skill boundaries: coordination relations with stable references and machine-checkable constraints become executable, while substantive task judgments remain within Skills. We introduce MetaSkill, a hybrid representation that preserves complete Skill packages and local instructions while making invocation dependencies and predecessor-text bindings executable through a directed acyclic graph (DAG) runtime. This interface supports explicit coordination without prescribing how each Skill performs its local task. Across 20 multi-Skill artifact tasks and two LLMs, we compare natural-language procedures, prompt-visible graphs, and MetaSkill derived from a common frozen plan per task, using the same designated Skill packages. Additional comparisons evaluate AgentSkillOS with the same available Skill packages as MetaSkill, as well as agent systems using their configured tools. Blinded artifact judgments yield favorable preference point estimates across all reported comparisons, with 95% task-bootstrap intervals excluding parity against Prompt Graph under DeepSeek-V4-Flash and OpenClaw under GLM-5.2. In audited GLM-5.2 execution traces, all planned nodes complete and all declared precedence constraints are satisfied. Together, these findings support MetaSkill as a practical interface for composing open-ended Skills, combining local linguistic flexibility with executable dependencies, explicit predecessor-text bindings, and auditable coordination. Code is available at: https://github.com/TokenRhythm/opensquilla/tree/paper/metaskill-lp4fm-2026