Identifying the Periodicity of Information in Natural Language

AI summary

Overview

Research area: Computational linguistics and information theory, specifically the study of how information (measured as surprisal) is distributed over time in natural language, with connections to discourse structure and LLM-generated text detection.

Technical level: Intermediate. The paper relies on Fourier analysis, periodograms, and autocorrelation functions, but the authors explain the machinery step by step and the core intuition can be followed without deep signal-processing background.

Scope: The paper introduces a document-level periodicity detection algorithm called AutoPeriod of Surprisal (APS), applies it across English and Chinese corpora, validates the results with harmonic regression, and uses it to compare human-written with LLM-generated text.

What This Paper Is About

Uniform Information Density (UID) theory predicts that information in language stays relatively constant, but this constancy is only an asymptotic property, so real texts inevitably show fluctuation. A stronger version of that fluctuation is periodicity: the possibility that information repeats in regular cycles, which earlier work could only detect indirectly through frequency-domain spectra or by validating periods that were already assumed to match known linguistic units. This paper asks whether periodicity can be detected directly at the level of a single document, with explicit confidence levels, and whether the detected periods map onto known linguistic structures.

Key Contributions

1. A new detection algorithm. The authors adapt the AutoPeriod algorithm (Vlachos et al., 2005) to surprisal sequences and release it as AutoPeriod of Surprisal (APS), a standalone toolkit that identifies periods directly in the time domain with defined confidence levels. Code is available at GitHub (CLCS-SUSTech/APS) and Zenodo (doi:10.5281/zenodo.19567886).

2. Empirical evidence for information periodicity. Applying APS to four corpora, the authors show that a non-negligible proportion of documents contain valid periods, with strict periodicity exceeding 5% of documents in every corpus and quasi-periodicity averaging about 15%.

3. Identification of periods beyond known structural units. A substantial share of detected periods are longer than typical sentences, paragraphs, and Elementary Discourse Units, indicating that periodicity is not fully explained by the succession of local discourse structures.

4. Independent validation and a human-versus-machine signal. The detected periods are confirmed as significant predictors in harmonic regression, and APS reveals that LLM-generated texts show stronger periodicity than human-written texts.

Main Findings

• Periodicity is detectable at the document level. Across the corpora examined, "a non-negligible proportion of texts exhibit strong periodicity of information." The strict periodic set P2 (documents with at least one valid period) reaches above 5% of all documents for every corpus.

• Most period hints survive validation. About 66% of documents with a period hint also contain a valid period, indicating that once GetPeriodHints detects a hint, it has a large chance of passing ACFFiltering.

• Chinese corpora show higher periodicity than English ones. All three ratios (P1/Σ, P2/P1, P2/Σ) are higher in Chinese than English. GCDT stands out with 30.00% of documents containing hints, 86.67% of those yielding valid periods, and 26.00% of all documents being strictly periodic, whereas WSJ**en shows 8.84%, 59.28%, and 5.24% respectively.

• Document counts and ratios. WSJen: 2,499 documents, 221 with hints, 131 with valid periods. Brownen: 500 / 52 / 25. GCDTzh: 50 / 15 / 13. CTBzh: 2,773 / 304 / 212. Averages across corpora: 15.05% for P1/Σ, 65.94% for P2/P1, 10.97% for P2/Σ.

• Detected periods partly align with structural units. The most prominent peak in the period distribution sits around 25 tokens, close to the peaks of EDU, sentence, and paragraph length distributions.

• But many periods are much longer than structural units. Only 3.63% of paragraphs exceed 150 tokens and only 1.11% exceed 200 tokens, while 33.33% of APS-valid periods exceed 150 tokens and 21.84% exceed 200 tokens. The authors conclude that global periodicity "is not fully explicable by the succession of local discourse structures alone."

• Harmonic regression confirms the detected periods. Using APS hints and valid periods as scaling factors yields significant harmonic coefficients. For valid periods on the WSJ P2 portion, the top amplitude is A1 = 0.2022 with β1,1 = 0.1213 and β2,1 = 0.1571, both with p = 0.000. For hints on the WSJ P1 portion, the top amplitude is A1 = 0.1772 with β1,1 = 0.0736 and β2,1 = 0.1612, both p = 0.000.

• Hints and valid periods behave almost equivalently in regression. There is no significant difference between their amplitude values, even though valid periods are stronger by the standard of passing the periodicity threshold. Their amplitudes are smaller than those reported for EDU in Tsipidi et al. (2025), which the authors attribute to hints and periods being fixed per document and therefore less position-dependent.

• APS-filtered portions are more predictable. Mean squared errors rank in the assumed order P2 < P1 < Σ < Σ − P1 across different scaling factors. On WSJ, EDU scaling gives 13.7788 for P2, 14.1434 for P1, 15.1843 for Σ, and 15.4160 for Σ − P1. On GCDT, EDU scaling gives 6.3902, 6.5097, 6.5961, and 6.5644 respectively. EDU scaling is the most effective, consistent with Tsipidi et al. (2025).

• LLM-generated text is more periodic. LLaMA3-70B-generated fake BBC news has 30.06% of documents with valid periods, more than twice the 14.80% in real news. LLM-generated texts show more periodic patterns especially in the long-period range (>50 tokens).

• Repetition explains only part of the gap. Sampling 10% of the documents with valid periods in LLaMA3-70B-generated BBC news (150 documents total), 7.33% (11 documents) contain repeated phrases or sentences. After excluding the repeated portion, the gap remains significant: 14.80% versus 27.86%.

• A worked example. For document #0976 from PTB-WSJ (TV listings), the most confident period hint at CL = .99 occurs at frequency f ≈ .019, corresponding to a period T ≈ 53.1 tokens. Raw hints of 53.2 and 177.2 are refined by ACF filtering to 52 and 163. A confidence level of .90 is used throughout the study.

Methodology in Plain English

The authors convert each document into a sequence of surprisal values, where surprisal is the negative log probability of each token given its preceding context, estimated by a language model. For English texts they use LLaMA3-8B, for Chinese texts Qwen2-7B, and for the RST Discourse Treebank they use Yarn-LLaMA2-7B to stay comparable with prior work.

APS then runs two steps. The first step, GetPeriodHints, computes a periodogram (a spectrum of the surprisal sequence) and compares the power at each frequency against a threshold derived from randomly shuffling the sequence 100 times and recording the maximum power each time. Periods whose power exceeds the 99th percentile of this random baseline are kept as "period hints" at a stated confidence level. In practice the authors use CL = .90. They chose the Lomb-Scargle periodogram variant to improve performance in the low-frequency (long-period) range.

The second step, ACFFiltering, uses the autocorrelation function, which measures how similar a sequence is to a shifted version of itself. A hint is accepted only if it falls on a hill of the ACF curve, and it is then refined to the nearest local maximum. Hints that do not sit on a hill are discarded as false alarms. The full version determines a search window from adjacent periods, finds the best split point via linear regression, and checks whether the split forms a hill using a slope-angle criterion (Δθ > 0.01).

For validation, the authors compute the autocorrelation function, which measures similarity between a sequence and a shifted copy of itself. Harmonic regression is used separately: they fit models that predict token surprisal from sine and cosine terms, using either their own detected periods or known structural units (EDUs, sentences, paragraphs) as the scaling factor, and check whether the harmonic coefficients are significant. They also compare the mean squared error of these models across differently filtered subsets of the data.

Why This Matters

Impact on research. The paper supplies a method that earlier work lacked: a way to detect periods directly rather than only validating periods selected from pre-defined linguistic units. It also shows that global periodicity and the local, structure-based periodicity studied by Tsipidi et al. (2025) are compatible rather than competing accounts, since APS-filtered portions are also better predicted by structural-unit scaling.

Real-world applications:

• LLM-generated text detection. Periodicity differs measurably between human- and machine-written text in both English and Chinese, making it a candidate feature for detection systems.
• Automatic NLG evaluation. Prior work uses surprisal spectra for generation quality assessment; APS gives an explicit, interpretable account of how generated text differs.
• Authorial and stylistic analysis. Because periodicity correlates with text genre and authorship-related factors, it could support stylometric profiling.
• Discourse and topic analysis. Longer periods that exceed paragraph length hint at latent structures such as topic spanning multiple paragraphs, which could inform discourse modeling.

Industry relevance. Any organization deploying or auditing large language models has an interest in cheap, interpretable signals that separate human from machine text. The toolkit is released publicly, so the method can be applied without reimplementing the algorithm.

Future Directions

• Multi-language extension. The study only examines English and Chinese, and the authors expect APS performance to vary across languages with different structural properties.

• Identifying the sources of long periods. The paper demonstrates that long periods exist but does not analyze what causes them. The authors speculate they may stem from latent structures such as topic spanning multiple paragraphs, and leave this to future work.

• Systematic study of text genre. The unusually high periodicity in GCDT (which spans academic articles, biographies, interview conversations, news, and how-to guides) versus the news-only corpora suggests genre is an important factor. The authors propose investigating dimensions such as verse versus prose and oral versus written as future work.

• Improving short-period detection. Frequencies returned by the discrete Fourier transform are unevenly distributed over period length, with short periods more densely packed, making them more prone to false positives under random perturbation. The authors note the short-period peak shrinks at higher confidence levels and that short-period detection is less robust.

Target Audience

This paper suits computational linguists and NLP researchers working on information theory, surprisal, and discourse structure, as well as researchers interested in LLM-generated text detection and evaluation. It is also relevant to signal-processing researchers curious about applications of periodicity detection to language data. Readers should be comfortable with basic probability and have some tolerance for Fourier and autocorrelation concepts, though the paper builds these up from the prerequisites rather than assuming prior exposure.

Note on reported details: The provided content is truncated in places. The detailed composition of the FACE and EvoBench subsets is said to be given in Table 8, but that table is not shown in the available text, and Table 7 (the remaining amplitudes and p-values for the full harmonic regression in Appendix B) is also cut off.

Authors’ abstract

Recent theoretical advancement of information density in natural language has brought the following question on desk: To what degree does natural language exhibit periodicity pattern in its encoded information? We address this question by introducing a new method called AutoPeriod of Surprisal (APS). APS adopts a canonical periodicity detection algorithm and is able to identify any significant periods that exist in the surprisal sequence of a single document. By applying the algorithm to a set of corpora, we have obtained the following interesting results: Firstly, a considerable proportion of human language demonstrates a strong pattern of periodicity in information; Secondly, new periods that are outside the distributions of typical structural units in text (e.g., sentence boundaries, elementary discourse units, etc.) are found and further confirmed via harmonic regression modeling. We conclude that the periodicity of information in language is a joint outcome from both structured factors and other driving factors that take effect at longer distances. The advantages of our periodicity detection method and its potentials in LLM-generation detection are further discussed.