Reinforcement Learning from Compiler and Language Server Feedback

AI summary

Overview

Research area: Natural Language Processing / code-generating language agents, with a systems and reinforcement-learning focus on how coding agents can be supervised by program-analysis tools.

Technical level: Intermediate. Readers benefit from some familiarity with the Language Server Protocol, reinforcement learning, and reward shaping, but the paper's motivation and design are explained in plain language.

Scope (one sentence): The paper proposes RLCSF, a method that turns compiler and language-server feedback into deterministic, replayable, transition-level rewards for coding agents, together with Lanser-CLI, a CLI-first orchestration layer that produces the artifacts those rewards are computed from.

What This Paper Is About

Coding agents that edit code often act on text-level guesses: they invent APIs that do not exist, resolve a name to the wrong symbol, or apply an edit that leaves the workspace in a worse state than before. Compilers, type checkers, and language servers already compute the missing facts (diagnostics, symbol resolution, type information, references, refactoring preconditions), but they expose them through interfaces built for human IDEs rather than for learning loops.

The paper's goal is to close that gap: make tool feedback dense, deterministic, and replayable, and convert it into a per-step reward signal an agent can plan or train against, instead of relying only on a terminal pass/fail signal.

Key Contributions

1. RLCSF (Reinforcement Learning from Compiler and Language Server Feedback): a shaped, process-level reward functional defined over adjacent Analysis Bundles, using diagnostic deltas, selector confidence, edit-safety readiness, and structured tool errors. It follows the potential-based reward-shaping template of Ng et al. (1999), with the potential grounded in machine-checked program facts.

2. Lanser-CLI: a CLI-first orchestration layer that mediates all agent–tool communication, manages the language-server lifecycle (start/stop, capability negotiation, cancellation, restart with backoff), coalesces identical in-flight queries via a single-flight cache, and exposes feedback as JSONL artifacts suitable for Unix tooling and CI.

3. Robust addressing via a Selector DSL: a PositionSpec tagged union (cursor, range, symbolic, AST path, content anchor) with a canonical string syntax, plus a Relocate algorithm that resolves possibly stale selectors, scores candidates deterministically, and surfaces ambiguity with ranked evidence instead of silently picking a target.

4. Determinism and safety guarantees: Analysis Bundles with pinned environment metadata and a SHA-256 hash (bundleId) over a JCS-canonicalized subset of fields, a proof that bundles are byte-stable under a frozen snapshot (Proposition 5.1), a proof that componentwise-improving transitions receive non-negative reward when undiscounted (Proposition 5.3), and a preview-first, jail-confined, Git-aware mutation workflow.

Main Findings

• Tool feedback can be turned into dense supervision. Rather than treating the compiler or language server as a terminal validator, RLCSF credits each intermediate action — locating a symbol, requesting diagnostics, previewing a rename, applying a guarded edit — with a transition-level reward suitable for planning, reinforcement learning, offline process supervision, and counterfactual evaluation.

• Determinism under frozen snapshots. Proposition 5.1 states that given a fixed workspace snapshot S, a fixed tool binary and configuration (V, Π), a negotiated positionEncoding, and a request Q, Lanser-CLI produces identical hash-domain canonical bundles across runs, so bundleId(B) is constant. Fields excluded from the hash domain, such as timestamps and run-local trace spans, are explicitly outside this claim.

• Non-negativity under componentwise improvement. Proposition 5.3 states that with γ = 1 and fixed non-negative weights, if diagnostics weakly decrease over the same scope (D_t ≤ D_t−1), safety readiness weakly improves (S_t ≥ S_t−1), selector confidence weakly improves (A_t ≥ A_t−1), and no structured tool error occurs (E_t = 0), then the reward r_t ≥ 0.

• Shaping is additive to task reward, not a replacement. When an external task reward exists, a learner can optimize R_task + r_csf. With w_E = 0 and bundle features in the Markov state, the shaping term reduces to the standard potential-based form; with w_E > 0 the tool-error penalty is an explicit design choice that may shift the optimal policy.

• Worked reward example (positive). With γ = 1 and weights (w_D, w_S, w_A, w_E) = (0.5, 0.4, 0.1, 0.5), a proposed rename from load_data to read_data that reduces Pyright diagnostics from 5 to 2, improves safety readiness from 0 to 1, improves selector confidence from 0.70 to 0.94, and raises no tool error yields r_csf = 1.924. The bundle records {"diag_delta": 3, "safety_delta": 1, "confidence_delta": 0.24, "tool_error": 0}.

• Worked reward example (negative). With the same weights, a refactor with unresolved imports where diagnostics stagnate (7 to 7), safety readiness does not improve (0 to 0), selector confidence stays at 0.62, and the preview reports E/APPLY_CONFLICT (E_t = 1) yields r_csf = −0.5, discouraging application until ambiguity and conflicts are resolved.

• Scoring for ambiguous selectors is a fixed convex combination. Non-certified candidates are ranked by score(s, c) = 0.5·s_ast + 0.2·s_module + 0.2·J_token + 0.1·s_prox, with all features normalized to [0, 1], inapplicable features set to 0, and the features recorded in the explanation. Exact snapshot maps and exact anchor-hash matches are certified with score 1.

• Rewards are reproducible offline. Because processReward is computed from adjacent canonical bundle contents and fixed weights — and is deliberately kept outside the current bundle's hash domain — the same r_csf is recovered by lanser trace replay, supporting offline evaluation and counterfactual policy analysis without re-running the language server.

• Self-hosting. Remark 2.1 notes that Lanser-CLI is used during its own development: to prepare and preview refactors in its repository, validate schema changes against historical traces, and replay bundles in CI to detect nondeterminism.

• Systems / benchmarks: The paper reports no empirical benchmark results, no dataset sizes, no model comparison tables, and no measured agent performance against baselines. It is a design-and-formalization paper; its evidence is the system design, the schema definitions, the two propositions with proof sketches, and the two worked reward examples.

Methodology in Plain English

The researchers start from the observation that the facts an agent needs are already computed by tools developers run every day, and that the obstacle is interface design rather than missing information.

Their approach has four parts. First, they build an orchestrator (Lanser-CLI) that sits between the agent and a pinned language server (they instantiate it against the Language Server Protocol using Pyright for Python) and turns each tool call into a structured Analysis Bundle — a JSON envelope with the request, resolution, facts, edits, environment metadata, capabilities, and reward.

Second, they replace fragile file:line:col references with a Selector DSL that describes intent (a symbol, an AST path, a content anchor, or explicit coordinates). When a selector may be stale, a Relocate algorithm resolves it against the current workspace, scores candidates with a fixed weighted formula, and either accepts the top candidate automatically (when it passes confidence and margin thresholds τ and δ) or surfaces ranked alternatives and demands explicit confirmation for mutations.

Third, they make output reproducible: lists are sorted by a fixed order over (uri, sL, sC, eL, eC), JSON is canonicalized with the JSON Canonicalization Scheme, environment fields such as tool version, server version, position encoding, Python executable and version, virtualenv path, config digest, and platform are pinned into the bundle, and the bundle ID is a SHA-256 hash over a canonicalized subset of fields that excludes volatile timestamps and run-local trace data.

Fourth, they define the reward. A potential Φ(B_t) = −w_D·D_t + w_S·S_t + w_A·A_t combines diagnostic count, safety readiness, and selector confidence; the reward is r_t = γ·Φ(B_t) − Φ(B_t−1) − w_E·E_t. Diagnostic deltas are only credited when the diagnostic scope is unchanged between steps; otherwise the bundle records scope_changed and the diagnostic component is zero unless a common scope is pinned. Read-only steps carry forward the previous safety value unless a safety check is observed. They then prove the two properties described above by proof sketch, since the underlying mechanics — deterministic sorting, canonicalization, and excluded volatile fields — make the arguments straightforward.

Safety is handled by making mutation preview-by-default: lanser rename requires an explicit --apply flag, edits are staged and written atomically per file via temp file plus rename(2) with fsync, multi-file edits are validated before any replacement with rollback metadata, conflicts are surfaced as E/APPLY_CONFLICT with machine-readable hunks, a realpath-normalized workspace jail confines writes to the project root, and a clean-Git-worktree guard can only be overridden with --allow-dirty.

Why This Matters

Impact on research. The paper reframes agent supervision as an artifact-design problem: if tool feedback is deterministic and hashable, rewards become replayable, auditable, and usable for offline process supervision and counterfactual policy analysis without re-running a language server. It also connects LLM-agent research to decades of existing program-analysis and refactoring infrastructure.

Real-world applications:

• Autonomous coding agents (the paper names GPT Codex and Claude Code as example language agents) that need grounded, per-step feedback rather than only final test results.
• CI/CD pipelines that gate refactors: lanser batch consumes JSONL command queues, lanser schema validates payloads before execution, and lanser trace replay regenerates byte-stable outputs to detect nondeterminism.
• Large-scale or multi-file refactoring, where preview-first renames, workspace jails, conflict reporting, and Git-backed rollback reduce the risk of partial or stray edits.
• RL training and offline evaluation of coding agents, using the transition reward for planning guidance where final success labels are sparse.

Industry relevance. The CLI-first, JSONL-serializable design composes with Unix tooling and containerizes cleanly, which is the shape most agent infrastructure and developer-tooling vendors already use. Pinning tool version, server version, encoding, interpreter, and config digest into each bundle gives teams a reproducible record for debugging across machines and for regression testing.

Future Directions

• Empirical validation. The paper reports no experiments, benchmarks, or dataset sizes; measuring whether RLCSF rewards actually improve agent repair and refactoring outcomes against strong baselines is the most obvious open question.
• Weight and threshold selection. The weights w_D, w_S, w_A, w_E and the selector thresholds τ and δ are treated as design choices. The paper notes that w_E > 0 may shift the optimal policy, but does not report how to set these values.
• Crash-atomicity for multi-file edits. The paper states explicitly that per-file atomic replacement via rename(2) does not, by itself, guarantee crash-atomicity for multi-file edits; whole-patch conflict detection and rollback can additionally be delegated to git apply --3way.
• Beyond the current instantiation. The system is instantiated against LSP with Pyright for Python. Extending it to other languages and to compiler-backed analyzers that emit the same bundle schema is suggested by the architecture but not demonstrated here.

Target Audience

Researchers and engineers building coding agents or agent-training pipelines; reinforcement-learning practitioners interested in reward shaping grounded in machine-checked signals; developer-tooling and platform engineers who want deterministic, audit-friendly artifacts from language servers; and program-analysis researchers interested in replayability, robust code addressing, and safe automated refactoring.

Authors’ abstract

Coding agents fail when text-level guesses outrun program facts: they hallucinate APIs, drift to the wrong symbol, and apply edits without evidence that the workspace remains valid. Compilers, type checkers, and language servers already compute the missing supervision signal, in the form of diagnostics, symbol resolution, type information, references, and refactoring preconditions, but expose it through interfaces designed for human-driven IDEs rather than learning loops. We introduce Reinforcement Learning from Compiler and Language Server Feedback (RLCSF) together with Lanser-CLI, a CLI-first orchestration layer that exposes this signal to agents and CI. RLCSF treats each tool interaction as a transition and computes a shaped process reward from deterministic changes in diagnostics, selector confidence, and edit safety. Lanser-CLI, in turn, converts ephemeral LSP sessions into replayable Analysis Bundles with pinned environment metadata and stable content hashes. Its core mechanisms are robust selectors that go beyond file:line:col, deterministic bundle normalization, preview-first guarded mutations, and a reward functional whose potential-based component is replayable under frozen snapshots. We formalize determinism for canonical bundles and prove that componentwise-improving transitions receive non-negative reward in the undiscounted setting. Together, these pieces yield a practical substrate for process supervision of coding agents.