How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction

AI summary

Overview

• Research area: Natural Language Processing — robustness and evaluation of large language models, specifically sensitivity to word/character order in Chinese, Japanese, and Korean four-character expressions.
• Technical level: Intermediate.
• Scope: The paper introduces OrderProbe, a deterministic multilingual benchmark of 3,543 curated four-character expressions, plus a six-metric diagnostic framework, and evaluates twelve LLMs on exact structural reconstruction from scrambled inputs.

What This Paper Is About

Sentence-level restoration is hard to score automatically because a scrambled sentence can often be reordered in several valid ways. The authors sidestep that ambiguity by using fixed four-character expressions in Chinese, Japanese, and Korean, which have a single conventional order and therefore allow exact-match scoring. They build a benchmark plus a diagnostic framework to measure how well LLMs can recover that canonical order from scrambled constituents — and to explain why recovery fails when it does.

Key Contributions

1. A deterministic target unit. The authors identify the ambiguity of sentence-level restoration and propose fixed four-character expressions as a deterministic proxy for evaluating structural reconstruction.
2. The OrderProbe benchmark. A multilingual benchmark containing 3,543 curated expressions across four script settings (Simplified Chinese, Traditional Chinese, Japanese, Korean) and six syntactic categories, with all 23 non-identity permutations generated per expression.
3. A diagnostic evaluation framework. Six metrics that go beyond recovery accuracy: Recovery Rate, Semantic Accuracy, Logical Validity, Structural Consistency, Robustness, and Information Density.
4. A twelve-model evaluation. Testing of twelve widely used LLMs showing that exact reconstruction remains difficult, with zero-shot recovery often below 35% even for strong systems.

Main Findings

• Zero-shot recovery is low across the board. Exact reconstruction is difficult even for strong systems, with zero-shot recovery frequently below 35%. The highest zero-shot Recovery Rate reported in Table 3 is 31.062 (Qwen-3-14B).
• A gap between meaning and structure. Models often generate fluent, semantically plausible explanations via lexical retrieval from partially preserved cues while still failing to reconstruct the canonical order. The paper describes this as a consistent gap between meaning-oriented generation and exact structural reconstruction.
• CoT helps, but unevenly. Chain-of-Thought prompting often improves recovery, but gains are strongly model-dependent and can reverse. DeepSeek-V3.2 rose from 20.104 to 62.238 (↑42.134), while Qwen-3-8B fell from 15.786 to 12.594 (↓3.192) and Qwen-3-14B improved only from 31.062 to 31.527 (↑0.465).
• Consistency is not correctness. High Structural Consistency can coincide with low recovery, because models may produce stable but generic explanations across permutations. Korean shows this most starkly: S_Cons of 0.7625 alongside a Recovery Rate of 5.72%.
• Script typology matters. Logographic scripts provide stronger local anchors: Simplified Chinese reaches 25.20% recovery, Traditional Chinese 20.32%, Japanese 19.66%, while Korean Hangul — treated as a phonogrammatic control — collapses to 5.72%.
• Permutation position matters. Recovery depends strongly on anchor displacement: permutations that keep semantically informative characters near their canonical neighborhoods yield higher recovery, while dispersed anchors degrade performance significantly.
• Few-shot is the most uniform. Three-shot in-context learning yields the most uniform improvements across permutations, whereas CoT gains are less stable and may amplify format drift in weaker models.
• Syntactic structure matters. Parallel and coordinate patterns are easiest due to redundancy and symmetric templates. Verb–object patterns are harder because directional dependencies are disrupted. For subject–predicate patterns, models often recover the underlying proposition but fail to linearize characters into the precise canonical order.
• Near-misses are common. Among non-exact zero-shot outputs, related but non-canonical expressions account for 12.80% and literal recombinations for 31.50%.
• Humans outperform models. Human participants recover 46.5–88.6% of the same scrambled inputs across scripts, while the canonical-input control reaches 100% Recovery by construction.
• Memorization alone does not explain the results. Recovery improves with idiom frequency, but failures remain common even in high-frequency bins. On a synthetic control set of non-attested four-character expressions, models continued to show low reconstruction accuracy, though slightly lower than on idiomatic items.
• The diagnostic pipeline is stable. Under controlled substitutions of the cross-encoder, embedding similarity, NLI model, aggregation strategy, and weight scheme, model rankings remained stable, with Spearman correlations ranging from 0.842 to 0.931 relative to the original configuration.

Methodology in Plain English

The benchmark is built in four stages.

1. Multi-source collection. Four-character expressions are aggregated from publicly available lexical dictionaries and digital repositories, yielding approximately 4,000 raw candidates, each tagged with a script typology (ZH-CN, ZH-TW, JA, KO).
2. Expert filtering. Senior linguists and trained annotators remove non-standard items, ambiguous items, modern internet slang, and items with multiple competing canonical forms, as well as expressions whose permutations could form another conventional lexicalized entry. Five native-speaker annotators handle each language independently, with majority vote and a consensus review for disputed cases. Inter-annotator agreement reaches Fleiss' κ = 0.835, and the process yields exactly 3,543 canonical expressions.
3. Semantic reference construction. Each expression gets a dictionary definition as its primary semantic anchor, augmented with LLM-generated paraphrastic variants verified by annotators. This produces a reference set of the dictionary definition plus augmented variants.
4. Permutation-based perturbation. Each expression is treated as a four-token sequence, and all 23 non-identity permutations are generated. This yields 81,489 perturbed inputs in total (3,543 × 23).

For evaluation, models receive a scrambled input and must output the recovered canonical expression plus a one-line semantic explanation in the matching language. Prompting is language-matched, using a unified two-message system-plus-user format with strict schema enforcement. Three prompting settings are compared: zero-shot, Chain-of-Thought (exactly two lines: reasoning plus final answer), and three-shot in-context learning. The twelve models span open-source checkpoints from Qwen, DeepSeek, Llama, and Gemma, and closed-source systems GPT, Gemini, Claude, and GLM.

Why This Matters

The paper argues that structural robustness is not an automatic byproduct of semantic competence — a model can explain what an expression means while being unable to reconstruct its order. This separates two capabilities that leaderboard scores often blur together, and it gives researchers a diagnostic signature per model rather than a single rank. The finding connects to prior work showing Transformers can be insensitive to word order and can fail to generalize relational structure under reversal.

Real-world applications:

• Search and query reconstruction. Systems that must repair scrambled or partially disordered user queries before retrieval.
• OCR and ASR post-processing. Recovering canonical word or character order from noisy recognition output where components survive but their sequence is corrupted.
• Low-resource and multilingual NLP. Language-matched evaluation across Chinese, Japanese, and Korean scripts helps identify where reconstruction depends on script-specific local anchors such as logographic characters.
• Model evaluation and selection. The six-metric framework distinguishes hallucinated retrieval, positional sensitivity, format mimicry, and verbose "knowledge dumping" as separate failure modes.

Industry relevance: organizations deploying LLMs in multilingual pipelines gain a concrete stress test for order sensitivity, and a way to check whether a model that answers fluently is actually reconstructing structure or merely retrieving memorized content.

Future Directions

• Broaden linguistic coverage. The authors state that OrderProbe is limited to six syntactic categories and fixed four-character expressions in three languages, and does not study alphabetic or morphologically rich systems in detail.
• Separate memorization from reconstruction. Broader tests on novel, synthetic, or non-idiomatic controls remain, since some expressions likely appear in pretraining corpora.
• Explain the CoT variability. Why Chain-of-Thought produces gains as large as ↑42.134 for one model and a regression of ↓3.192 for another remains an open question.
• Interpret consistency metrics jointly. The Korean result — high consistency with 5.72% recovery — suggests the need for methods that distinguish genuine invariance from stable generic output under uncertainty.

Target Audience

Researchers and engineers working on LLM robustness, evaluation benchmark design, multilingual NLP for Chinese, Japanese, and Korean, and idiom or figurative-language processing. It is also relevant to practitioners who deploy multilingual models and need to know whether fluent output reflects genuine structural understanding or retrieval alone.

Authors’ abstract

Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored. Sentence-level restoration is difficult to evaluate automatically because scrambled sentences often admit multiple valid reorderings. We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring. We further propose a diagnostic framework that evaluates models beyond recovery accuracy, including Semantic Accuracy, Logical Validity, Structural Consistency, Robustness, and Information Density. Experiments on twelve widely used LLMs show that structural reconstruction remains difficult even for frontier systems: zero-shot recovery frequently falls below 35%. We also observe a consistent gap between meaning-oriented generation and exact structural reconstruction, suggesting that structural robustness is not an automatic byproduct of semantic competence.