String Seed of Thought: Prompting LLMs for Distribution-Faithful and Diverse Generation

AI summary

Overview

• Research area: Large language model prompting, probabilistic instruction following, and output diversity for generative AI.
• Technical level: Intermediate. The paper combines practical prompting recipes with statistical theory (total variation distance, 2-universal hashing), but the core method is a plain-language prompt instruction.
• Scope: The paper proposes String Seed of Thought (SSoT), a single prompting technique that makes LLMs sample from specified probability distributions and produce more diverse outputs, and evaluates it across five frontier LLMs, a Rock-Paper-Scissors adversarial testbed, and NoveltyBench.

What This Paper Is About

LLMs are good at questions with one correct answer, but they are often biased when asked to choose among several valid answers according to a target probability distribution (for example, "flip a fair coin, heads or tails, 50-50"). Repeatedly prompting the same model can produce skewed, collapsed outputs, which hurts applications such as simulating human behavior, diversifying content, and playing mixed-strategy games.

The paper's goal is to fix this without fine-tuning: it introduces SSoT, which tells the LLM to first generate a random string and then manipulate that string to derive its answer, so the choice becomes a nearly uniform random function of the string rather than a biased direct pick. The authors define this setting as Probabilistic Instruction Following (PIF), contrast it with Diversity-Aware Generation (DAG), and test SSoT in both.

Key Contributions

1. The SSoT prompt. A simple prompting technique that steers an LLM's probabilistic behavior by having it internally generate a random string and process that string to reach a decision. It works in the system prompt or the user prompt, and applies without retraining or tools.
2. Empirical results in two regimes. The paper reports that SSoT achieves sampling faithfulness in PIF comparable to a pseudo-random number generator (PRNG), and that it boosts response diversity in DAG without hurting quality across the tested benchmarks.
3. A theoretical guarantee. The authors prove that the total variation distance between the empirical distribution produced by SSoT and the target distribution shrinks as the generated string gets longer, even when the string's characters are autoregressively correlated.
4. Mechanistic analysis. The paper shows that LLMs autonomously choose their own randomness-extraction strategies (Sum-Mod versus Rolling Hash, depending on task difficulty), and that PIF performance scales with the length of the model's chain of thought.

Main Findings

• PIF improvement across five frontier LLMs. Using K = 100 trials per experiment, repeated 10 times, SSoT reduced JS divergence over baseline prompting for deepseek-v3-0324 (T = 1.0), gpt-4o-2024-08-06 (T = 1.0), o4-mini-high (temperature unavailable), QwQ-32B (T = 0.6), and deepseek-r1-0528 (T = 0.6), across 2-choice, biased 2-choice, 3-choice, biased 3-choice, and biased 9-choice settings.
• Near-PRNG behavior for long-reasoning models. For example, deepseek-r1 on 2-choice went from a baseline JS divergence of 36.09 ± 13.60 to 3.03 ± 3.43 with SSoT (a reported 92% reduction), close to the numpy.random PRNG reference of 1.85 ± 2.58. QwQ-32B on biased 9-choice went from 260.59 ± 18.95 to 11.48 ± 5.27 (96% reduction), against a PRNG reference of 13.72 ± 4.21.
• One reported exception. QwQ-32B on plain 2-choice was slightly worse with SSoT (3.39 ± 4.97) than its baseline (2.43 ± 4.57), a 40% increase; the paper attributes a related failure case to a "lazy" strategy that ignores most of the string's entropy.
• SSoT beats other prompting baselines on wider action spaces. Tested with deepseek-r1 on 2^n actions for n = 1 to 6, using action words drawn from a list of the 10,000 most common English words, SSoT was best in both uniform and biased conditions and nearly matched the PRNG. Its strongest competitor, Prompt Ensemble, did well when unbiased but degraded on biased tasks.
• Biased distributions are harder than uniform ones. The paper argues this is because biased PIF requires both avoiding mode collapse and deriving plus executing correct arithmetic to match the target probabilities, which uniform sampling does not require.
• External randomness helps but is not as good. An ablation giving the model an externally supplied random string (Fixed or Randomized) improved performance, confirming LLMs can select actions given a randomness source; SSoT still outperformed external randomized seeds in the biased setting.
• Wins in an adversarial game. Against 10 "Black Belt" bots from an RPS Dojo Kaggle notebook, over 100 consecutive games per match with +1 per win and -1 per loss, SSoT held an average score near zero while the bots had access to the full move history and the LLM did not. A Baseline prompt showed exploitable biases and a Simple prompt was consistently defeated.
• Diversity gains without quality loss. On NoveltyBench with eight responses per question using deepseek-r1 (T = 0.6), SSoT reached Overall Distinct 6.19 (Utility 5.92) on the curated dataset versus 4.70 (5.17) for the baseline, and 5.25 (4.86) versus 3.39 (4.08) on WildChat. On the "Creativity" category, SSoT reached Distinct 5.90 and Utility 6.44, beating the compared methods on both.
• The model invents its own strategy. Analyzing 600 responses each from the biased and unbiased PIF experiments with gemini-flash-2.5, the CoTs converged on Sum-Mod (sum character codes, take modulo) for unbiased tasks and Rolling Hash (polynomial hash with base B, then modulo) for biased tasks.
• Diversity comes from decomposition in creative tasks. Across all 800 responses on the NoveltyBench curated dataset, the general pattern was choosing from a fixed List with one Global sample, but for "Creativity" it reversed: the model built a Template and resampled from the random string for each Local element.
• Performance scales with reasoning length. With s1.1-32B and budget forcing, generated integers between 0 and 127 became more uniform as thinking tokens grew (N = 1000, measured by KL/JS divergence and effect size w). String complexity also grew with thinking length: measured by normalized Lempel-Ziv complexity and zlib compression rate at T = 0, using the 3000-character prefix of 100 generated strings.

Methodology in Plain English

The method is a two-step instruction added to the prompt. First, the LLM writes out a random string inside <random_string> tags. Second, it reasons inside <thinking> tags and is told to use all the content of that string to make any probabilistic decision, before giving its final answer in <answer> tags. For PIF the instruction is "generate a random string, and manipulate it to sample from the target distribution"; for DAG it is "generate a random string, and manipulate it to generate one diverse response."

The intuition is that the simple, task-agnostic act of string generation is less exposed to the biases (option ordering, label frequency in training data) that contaminate direct action selection, and that mapping the string to an action can be done with arithmetic an LLM already knows. Because each generation is independent, the approach is fully parallelizable, unlike sequential sampling that needs generation history.

Evaluation repeats the same prompt K times and parses the outputs into actions, then compares the empirical action distribution against the target using total variation distance, KL divergence, and JS divergence. The authors also compare against an ideal reference built with numpy.random (the PRNG row), test five models, sweep action-set sizes, benchmark against High Temperature, Few-shot (k in {3, 10, 50}), Prompt Ensemble (50 paraphrased prompts with randomized action orders), and Sequential Sampling, run an adversarial Rock-Paper-Scissors evaluation, and use NoveltyBench for open-ended diversity. The theory side proves two bounds: one for extracting randomness with a random 2-universal hash function (Theorem 4.1) and one for the sum-mod strategy observed in the models' chains of thought (Theorem 4.2). Both bounds contain a term that shrinks with string length n and a finite-sample term that shrinks with the number of samples K. Biased target distributions are handled in Appendix G.6.

Why This Matters

• Impact on research: The work reframes distributional alignment as a prompting problem rather than a fine-tuning problem, offers a theoretical link between generated string length and sampling fidelity, and documents concretely which internal strategies LLMs invent to extract randomness — a level of mechanistic detail that prior distributional-alignment work (for example Meister et al., 2025; Gu et al., 2025; and studies of coin-flip bias) generally did not provide.
• Human-behavior simulation: Representing collective opinion distributions requires sampling views in the right proportions, which is exactly the PIF setting the paper targets.
• Game-playing and security: Mixed-strategy Nash equilibria depend on being unpredictable; the Rock-Paper-Scissors experiment shows a non-exploitable defense that needs no external tooling.
• Content diversification and brainstorming: Open-ended tasks such as naming, story writing, and idea generation benefit from the DAG-side diversity gains demonstrated on NoveltyBench.
• Test-time scaling: Methods that generate many candidate solutions and pick the best rely on candidate diversity, so reducing collapse into a small answer set supports these pipelines.
• Industry relevance: The technique is tuning-free, needs only a prompt change, works on reasoning models where the temperature parameter cannot be modified (such as o4-mini-high), and is fully parallelizable, which makes it cheap to adopt relative to reinforcement-learning or preference-optimization approaches for diversity.

Future Directions

• Extending to harder models and settings. The paper notes SSoT is most effective on highly capable models and suggests applying it to more complex domains; models with limited reasoning capability (typically under 8B parameters) may fail to execute the mapping arithmetic.
• Hardening against bias propagation. When the generated string has strong positional bias and the model adopts a "lazy" strategy (for example, using only the first character), output distributions become skewed. The authors suggest system-prompt steering toward robust strategies such as rolling hashes.
• Optimizing seed generation and extraction. The conclusion explicitly lists improving the string seed generation and the randomness-extraction strategy as promising avenues.
• Scoping the method correctly. The limitations section leaves open how to handle single-answer tasks, where SSoT is described as not effective and potentially distracting; the paper notes utility was generally preserved on NoveltyBench but does not settle the question.

Target Audience

Researchers and engineers working on LLM behavior control, distributional alignment, and diversity of generated text; practitioners building human-behavior simulators, game agents, or content-generation systems who need models to follow stated probabilities rather than their own priors; and theory-minded readers interested in how randomness extraction bounds carry over from hashing and random-walk analysis to LLM prompting. The paper assumes familiarity with metrics such as total variation distance and JS divergence, so readers without a statistics background will find the experimental results easier to follow than the theoretical sections.

Authors’ abstract

We introduce String Seed of Thought (SSoT), a novel prompting method for LLMs that improves Probabilistic Instruction Following (PIF). We define PIF as a task requiring an LLM to select its answer from a predefined set of options, each associated with a specific probability, such that the empirical distribution of the generated answers aligns with the target distribution when prompted multiple times. While LLMs excel at tasks with single, deterministic answers, they often fail at PIF, exhibiting biases problematic for applications requiring non-deterministic behaviors, such as human-behavior simulation, content diversification, and multiplayer games. It also harms the diversity of generated responses, a crucial factor in test-time scaling, by causing the outputs to collapse into a limited set of answers. To address this, we propose SSoT, a simple prompting method that instructs an LLM to first output a random string to generate sufficient entropy. SSoT also instructs the LLM to extract randomness by manipulating this string to derive a final answer, thereby preserving diversity while adhering to specific constraints. We demonstrate that SSoT significantly improves the PIF performance of LLMs, approaching the ideal performance of a pseudo-random number generator. Furthermore, our experiments on NoveltyBench show SSoT's benefits extend beyond closed-set tasks to open-ended tasks by enhancing response diversity.