BoundRL: Efficient Structured Text Segmentation through Reinforced Boundary Generation

AI summary

Overview

Research area: Natural Language Processing — text segmentation, large language model fine-tuning, and reinforcement learning with verifiable rewards (RLVR).

Technical level: Intermediate. Readers should be comfortable with the broad ideas of supervised fine-tuning, reward-based reinforcement learning, and evaluation metrics such as F1 and boundary-detection scores.

Scope: The paper introduces BoundRL, a method that recasts token-level segmentation of structured text (LLM prompts containing code, JSON, tables, and placeholders) as a short "boundary generation" problem, and trains small language models to solve it with reinforcement learning.

What This Paper Is About

Conventional text segmentation splits documents at sentence or paragraph boundaries, but many structured texts — especially LLM prompts — mix natural-language instructions with code snippets, JSON, and placeholders that do not follow those boundaries. Prior LLM-based approaches segment such text by regenerating the full text of every segment, which is expensive and prone to hallucination. BoundRL instead generates only a short sequence of starting tokens (plus a label) for each segment and reconstructs the full segment text by locating those tokens inside the original input.

Key Contributions

1. BoundRL, a boundary-generation formulation for token-level structured text segmentation that reduces output tokens by 90% and mitigates hallucination risks, shifting output complexity from O(|d|) (input length) to O(n) (number of segments).
2. A dual-objective reward function for RLVR that jointly optimizes reconstruction fidelity (measured by a reconstruction ratio) and semantic alignment (measured by exact-match F1 and character-level F1).
3. A selective perturbation strategy that builds intermediate candidates by adjusting boundaries and labels, addressing entropy collapse in RLVR by creating learnable stepping stones between the model's current generations and the annotated segments.
4. StructSeg, a 15.3K human-annotated benchmark of synthetic prompts and prompts from LangSmith, on which BoundRL enables 1.7B-parameter models to outperform much larger models such as Claude-4 Sonnet.

Main Findings

• BoundRL beats all baselines consistently: Across Qwen3-1.7b, Qwen3-4b, and Llama-3.1-8b-Instruct, BoundRL achieved the best results. The gap between BoundRL and the second-best method (SFT+RLVR) was statistically significant using a paired t-test (p < 0.05).
• Small models beat large prompted models: Qwen3-1.7b trained with BoundRL reached an average score of 84.5, versus the best few-shot prompting baseline (Claude4-Sonnet with the "full" output pattern) at 68.8 and Claude4-Sonnet with the "start" pattern at 68.8. Claude3.5-Sonnet full scored 58.2.
• Token efficiency: Prompting baselines that generate full segment text required an average of 1,170 output tokens per input prompt on the Synthetic subset, while BoundRL required only 119 — a 90% reduction.
• RLVR beats SFT, especially out of domain: Models fine-tuned with RLVR consistently outperformed SFT-only models, and the difference between SFT+RLVR and SFT w/2epochs was statistically significant (p < 0.05). Improvements were larger on the Langchain subset, indicating better generalization to real-world, out-of-domain prompts. For example, Qwen3-1.7b's Langchain exact-match score rose from 34.7 with SFT to 47.3 with BoundRL.
• Why SFT is insufficient: A substantial share of SFT predictions identified the correct boundary positions but used lexically different starting tokens — 21% for Llama-3.1-8b, 28% for Qwen3-1.7b, and 19% for Qwen3-4b. SFT wrongly penalizes such tokens, whereas the reward design gives identical rewards to different starting tokens that correspond to the same boundary.
• Annotated segments are not useful during rollout: RL-PLUS, which uses annotated segments during rollout, produced inconsistent results and could even hurt performance, suggesting annotated segments are too out-of-distribution to provide useful learning signals.
• Higher temperature does not help: SFT+RLVR with a sampling temperature of 1.5 during rollout did not improve performance, indicating the gain from intermediate candidates comes from guided exploration rather than simply a larger exploration space.
• Sentence-level segmentation is inadequate: The Oracle sent baseline, which has access to gold annotations but maps sentences to the most overlapping label, scored poorly on exact match and P_k (an average of 75.4 versus 84.5 for BoundRL on Qwen3-1.7b).
• Token-level NER is weak on boundaries: Models fine-tuned for NER achieved high character-level F1 but lower scores on exact match, P_k, and label F1, and tended to generate fragmented, short segments.
• Ablations confirm design choices: BoundRL outperformed all ablated versions. Using two perturbation steps, replacing candidates for all inputs without selective replacement, and perturbing a randomly sampled candidate instead of the medium-reward one all hurt performance.
• Starting tokens are the best output pattern: Averaged over both subsets, Qwen3-1.7b scored 81.1 with "start", 75.6 with "end", and 74.8 with "start+end"; Qwen3-4b scored 83.5 / 82.3 / 80.3; Llama-3.1-8b-Instruct scored 82.5 / 80.9 / 77.6. Generating both boundaries imposed too much learning burden.
• Sentence-level oracle ceiling: Oracle sent reached a Synthetic reconstruction ratio of 100.0 but only 2.7 exact match and P_k of 16.5, showing prior sentence-level methods are fundamentally unsuited to structured text.
• Additional dataset: The paper states that BoundRL was also evaluated on the Wikisection-city dataset; those results appear in Appendix A.9, which is not included in the provided content.

Methodology in Plain English

BoundRL trains a language model in two stages.

Stage 1 — Supervised fine-tuning (SFT). The annotated segments are converted into short "starting token" sequences, randomly sampled between 2 and 10 tokens long and lengthened when two segments would otherwise share the same starting tokens. The model is fine-tuned to output a list of (label, starting tokens) pairs. At inference, segment texts are recovered by locating each starting-token sequence in the input, always taking the leftmost occurrence after the previous segment's position. This ordering constraint lets repeated token sequences be assigned uniquely to segments. If either a segment's starting tokens or the next segment's starting tokens cannot be found, that segment is discarded.

Stage 2 — Reinforcement learning with verifiable rewards (RLVR). Rewards are rule-based and combine two dimensions:

• Reconstruction fidelity: the reconstruction ratio, the proportion of input characters that can be recovered from the generated segments.
• Semantic alignment: the F1 of exact match between generated and annotated segments, plus a character-level F1 that treats segmentation as assigning each character a label.

The final reward multiplies the reconstruction ratio by the average of the two alignment scores.

Intermediate candidates. During rollout, the model generates m candidate segmentations per input, ordered by reward. BoundRL perturbs the candidate with the medium-level reward by shortening or extending a segment by one word on either side, or by swapping a label for an alternative not already used by neighboring segments. The best-scoring perturbed version becomes the intermediate candidate. To avoid off-policy harm, replacement is selective: an intermediate candidate is used only if its reward exceeds the original, and only for at most k inputs per batch, choosing the top-k by gain.

Data and models. StructSeg contains synthetic prompts generated with Claude 3.5 Sonnet plus real-world prompts from the Langchain hub. Each segment carries one of five labels: instruction, example, context, question, or output format. Table 1 reports 15,132 synthetic prompts and 197 Langchain prompts, with token and segment statistics of 900 tokens / 6.1 segments for Synthetic and 914 tokens / 7.6 segments for Langchain. Some prompts exceed 2,000 words. The Synthetic subset is split into 14,732 training, 200 validation, and 200 test prompts; the Langchain prompts are used exclusively for testing. Training used Qwen3-1.7b, Qwen3-4b, and Llama-3.1-8b-Instruct; SFT ran one epoch with batch size 16 (learning rates 2e-6 for Qwen3 and 5e-7 for Llama), and RLVR used GRPO on a randomly sampled 25% of training data with 6 documents per batch, m = 4 candidates, rollout temperature 1.2, k = 2 for the Qwen3 models and k = 1 for Llama. Human annotation achieved 89% inter-annotator agreement.

Why This Matters

Impact on research: The paper challenges the assumption that segmentation must operate at sentence or paragraph level, and shows that a generation-style output pattern can be made both cheaper and more accurate by outputting only boundaries. It also offers a concrete recipe for mitigating entropy collapse in RLVR using in-distribution intermediate candidates, which may transfer to other tasks with dense, continuous rewards.

Real-world applications:

• LLM prompt analysis and optimization: breaking long prompts into labeled instructions, examples, context, questions, and output formats.
• Retrieval-augmented QA: retrieving finer-grained, more relevant contexts from documents containing tables, code, and placeholders.
• Summarization of long documents with mixed structure, where segment-level structure guides the summary.
• Document processing in domains with heavy structure, such as legal documents and technical specifications, which the authors name as a possible extension.

Industry relevance: The method lets small models (1.7B-4B parameters) match or exceed few-shot prompting with far larger proprietary models on this task while producing 90% fewer output tokens, which translates directly into lower inference cost and latency.

Future Directions

• Reducing reliance on domain-specific annotated data, since the paper's Limitations section notes the method still requires annotated datasets and task-specific fine-tuning and does not work in true zero-shot settings.
• Exploring data-efficient training strategies such as improved annotation reuse or weak and semi-supervised learning.
• Extending the boundary-generation paradigm to other structured text domains, such

Authors’ abstract

Structured texts refer to texts containing structured elements beyond plain texts, such as code snippets and placeholders. Such structured texts increasingly require segmentation into semantically meaningful components, which cannot be effectively handled by conventional sentence-level segmentation methods. To address this, we propose BoundRL, a novel approach that jointly performs efficient token-level text segmentation and label prediction for long structured texts. Instead of generating full texts for each segment, it generates only starting tokens and reconstructs the complete texts by locating these tokens within the original texts, thereby reducing output tokens by 90% and minimizing hallucination. To train the models for the boundary generation, BoundRL~performs reinforcement learning with verifiable rewards (RLVR) that jointly optimizes document reconstruction fidelity and semantic alignment. It further mitigates entropy collapse by constructing intermediate candidates by perturbing segment boundaries and labels to create stepping stones toward higher-quality solutions. Experiments show that BoundRL enables small language models (1.7B parameters) to outperform few-shot prompting with much larger models as well as SFT and standard RLVR baselines on complex prompts used for LLM applications.