Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment

AI summary

Overview

• Research area: Large language model alignment, reinforcement learning from AI feedback, and scalable self-improving training methods.
• Technical level: Intermediate (familiarity with reinforcement learning for LLMs, reward models, and GRPO helps, but the core argument is conceptual).
• Scope: The paper proposes an open-ended alignment value called Collective Agency (CA) and a self-improving framework called Dynamic Alignment, then tests it by fine-tuning gpt-oss-20b and comparing it against its base version on a CA evaluation set and three general NLP benchmarks.

What This Paper Is About

Most LLM alignment targets fixed human-centric values such as helpfulness, honesty, and harmlessness, and relies on human preference data that is expensive and slow to scale. The authors argue these static value systems may be insufficient for systems approaching AGI or ASI, and that compressing diverse human values into one objective risks epistemic capture, where a single perspective dominates. They propose an open-ended value, Collective Agency, plus a framework that lets a model generate its own training data and reward itself toward that value, aiming to scale alignment beyond human feedback.

Key Contributions

1. Introduces Collective Agency (CA) as a unified, open-ended alignment value defined as the "infinite expansion of agency across spacetime," structured around four inseparable aspects: Knowledge, Benevolence, Power, and Vitality.
2. Proposes the Dynamic Alignment framework, which combines automated training dataset generation with a self-rewarding mechanism, allowing a model to iteratively align itself to CA without human-labeled data.
3. Builds a multi-model data generation pipeline that produced 1,000 unique task prompts designed to elicit CA-relevant behavior without naming CA or its four aspects, with a generator–evaluator–refiner loop.
4. Demonstrates the approach empirically by fine-tuning gpt-oss-20b with GRPO, showing substantially higher preference for the CA-aligned model on a CA evaluation set while general NLP benchmark performance remains statistically equivalent to the base model.

Main Findings

• Large CA alignment gain: On a held-out CA evaluation set of 100 prompts judged by GPT-4.1, the CA-aligned gpt-oss-20b won 87.2 (SD ± 4.10) percent of comparisons against the base model, which won 12.8 (SD ± 4.10) percent.
• General capabilities preserved: IFEval accuracy was 73.0 (SD ± 0.75) for the base model versus 72.4 (SD ± 1.08) for the CA-aligned model across 541 samples; GPQA Diamond was 37.0 (SD ± 2.97) versus 36.6 (SD ± 1.61) across 198 samples; AIME 2025 was 34.4 (SD ± 3.85) versus 34.4 (SD ± 6.94) across 30 samples. The latter two comparisons are marked as statistically equivalent under a paired bootstrap test with 95% confidence interval and 5% and 10% equivalence margins respectively.
• What the judge preferred: According to the GPT-4.1 judge's explanations, preferred CA-aligned responses showed deeper benevolence and vitality more consistently and wove all four CA aspects together more cohesively.
• Single holistic score, not per-aspect scores: The self-reward assigns one unified CA score on a 0–5 rubric rather than separate scores per aspect, which the authors describe as consistent with CA being an entangled structure.
• Prompt artifacts excluded from learning: Output candidates were generated using a system prompt that introduces CA, but that system prompt was excluded from the gradient update to avoid overfitting to prompt artifacts.
• Generated data diversity: Diversity of the generated training prompts is reported as pairwise similarity measured with ROUGE-L along with length distribution, though the paper does not report specific numeric values for these in the provided content.
• Prototype scope: The reported framework is a prototype implementation applied in a static dataset setting, evaluated with a single LLM judge.

Methodology in Plain English

The authors first define Collective Agency: rather than telling a model to produce specific behaviors, they ask it to advance four connected capacities at once, namely understanding (Knowledge), the ability to act (Power), the ability to renew and endure (Vitality), and commitment to empowering others (Benevolence). Because progress in one is assumed to require the others, the authors argue superficial or deceptive strategies are less likely to score well.

To build training data, they used OpenAI's o4-mini (2025-04-16) in a three-role pipeline. One model generates everyday goals that involve planning or decision-making, keeping prior outputs in context so successive goals stay diverse. A second model turns each goal into a task prompt that must be open-ended, must not mention CA or its aspects, must not specify factors to consider or expected outcomes, must avoid narrow NLP or domain-specific tasks, must avoid strictly structured output formats, and must stay within 100 words. A third model acts as an evaluator, judging each prompt as Appropriate or Inappropriate; if it fails, the critique is sent back for revision. This loop yielded 1,000 prompts.

For self-improvement, each prompt is given to the policy model, which produces G = 8 candidate responses under a system prompt describing CA. The same model then scores each candidate on how much it advances CA, using a 0–5 rubric (0 constrains CA; 1 neutral or irrelevant; 2 narrow advance of one aspect; 3 partial advance of multiple aspects; 4 strong balanced advance of at least three aspects; 5 transformative holistic advance of all four). Those scores feed Group Relative Policy Optimization (GRPO; Shao et al. 2024), which computes advantages and a loss used to update the model.

Training details: gpt-oss-20b was fine-tuned with batch size 32 and learning rate 5.0 × 10⁻⁶ in low reasoning mode, using GRPO group size G = 8, clipping threshold ε = 0.2, and entropy coefficient β = 0.04. Output candidates were generated with top-p sampling at p = 1.0 and T = 1.0, while sampling was disabled for the self-reward step to keep scoring consistent. Training ran until reward converged on a single NVIDIA H100 NVL GPU with 94GB of VRAM.

Evaluation compared the base and CA-aligned models on 100 held-out prompts, with GPT-4.1 as judge. To avoid positional bias, each prompt was judged twice with the aligned model's output in first and then second position, and a win counted only if the aligned model was preferred in both; inconsistent judgments were excluded from the win-rate calculation. General capabilities were measured on IFEval (prompt-level strict accuracy using the official evaluation codebase), AIME 2025, and GPQA Diamond (the last two scored by GPT-4.1 as an automated evaluator checking answer match and expected solution format). Reported averages and standard deviations come from repeating the evaluations three times.

Why This Matters

• Research impact: The paper shifts alignment away from static value lists toward an open-ended directional principle, and pairs it with a fully self-improving reinforcement learning setup rather than the preference-pair construction used in earlier self-rewarding work. It also extends the use of GRPO beyond conventional human-defined objectives.
• Real-world applications (framed by the paper's own motivation):
  • Planning and decision-support tasks, such as the family-vacation itinerary example the authors use to illustrate their generated data.
  • Reducing dependence on human annotation pipelines for alignment, since the framework generates its own task prompts and rewards.
  • Instruction-following, mathematical reasoning, and scientific question-answering use cases, which the authors benchmark to show no regression.
  • Future settings involving negotiation among agents with differing objectives, which the authors list as planned multi-agent work rather than something already demonstrated.
• Industry relevance: The approach targets the cost and iteration-speed bottleneck of RLHF for increasingly large models, offering a route to alignment updates that do not require fresh human preference labels. The reported training footprint is a single NVIDIA H100 NVL GPU with 94GB of VRAM on a 20-billion-parameter model, which is relatively accessible compared with large-scale RLHF pipelines.
• Caveat on evidence: The paper reports no deployment results, no human-evaluator comparison, and no comparison against other self-improving alignment methods in the experiments presented.

Future Directions

1. Multi-agent alignment: Extend the framework so model instances with differing objectives negotiate mutually agreeable solutions, adding multi-perspective oversight and reducing the risk of self-reinforced value drift in a single-agent loop.
2. Interpretability and operationalization of CA: Decompose the single holistic CA score into its four constituent pillars and build evaluation suites that give granular feedback, plus expand evaluation to multi-model and human feedback to reduce reliance on a single LLM judge.
3. Dynamic curriculum: Introduce dynamically generated tasks of increasing complexity to support continual capability growth, matching CA's expansive framing.
4. Broader benchmarking and ablations: Compare directly against alternative self-improving methods such as Constitutional AI and prior self-rewarding work, test across diverse architectures and training scales, isolate each component's contribution, and check whether CA alignment causes regressions on standard goals such as harmlessness and honesty.

Target Audience

Researchers and engineers working on LLM alignment, reinforcement learning from AI feedback, and self-improving training pipelines; safety and policy researchers interested in open-ended alignment objectives and reward hacking; and practitioners who need scalable alignment methods that avoid continued human annotation. Readers looking for large-scale deployment evidence or head-to-head comparisons against established alignment methods will find those are explicitly deferred to future work.

Authors’ abstract

Large Language Models (LLMs) are typically aligned with human values using preference data or predefined principles such as helpfulness, honesty, and harmlessness. However, as AI systems progress toward Artificial General Intelligence (AGI) and Artificial Superintelligence (ASI), such value systems may become insufficient. In addition, human feedback-based alignment remains resource-intensive and difficult to scale. While AI-feedback-based self-improving alignment methods have been explored as a scalable alternative, they have largely remained constrained to conventional alignment values. In this work, we explore both a more holistic alignment objective and a scalable, self-improving alignment approach. Aiming to transcend conventional alignment norms, we introduce Collective Agency (CA)-a unified and open-ended alignment value that encourages integrated agentic capabilities. We also propose Dynamic Alignment-an alignment framework that enables an LLM to iteratively align itself. Dynamic Alignment comprises two key components: (1) automated training dataset generation with LLMs, and (2) a self-rewarding mechanism, where the policy model evaluates its own output candidates and assigns rewards for GRPO-based learning. Experimental results demonstrate that our approach successfully aligns the model to CA while preserving general NLP capabilities.