Summary
Can the fluctuations of introductory text be measured like water flow? This study, published on September 27, 2026 in Entropy, transforms articles into high‑dimensional trajectories within language models and finds that both human‑written and AI‑generated texts exhibit …
Can the undulations of text be measured like water flow? This study, published on September 27, 2026 in Entropy , transforms articles into high‑dimensional trajectories within language models and finds that the power spectra of contextual representations in both human‑written and AI‑generated text exhibit a scaling close to 5/3— a number that also appears in classic turbulence theory. When word order is scrambled or static word vectors without context are used, this pattern disappears. While this is not enough to prove that language is turbulence, it provides a new ruler for observing the multi‑scale order deep within language.
Keywords: language models, contextual representations, 5/3 scaling law, power spectrum, multi‑scale structure, turbulence analogy
Jizhi Editorial Team | Author
Paper title: Empirical 5/3 Spectral Scaling in Contextual Representations of Language
Paper link: https://www.mdpi.com/1099-4300/28/10/1063
Published: September 27, 2026
If I told you that the line you’re reading now and the frothy foam swirling in a cup of coffee might obey the same mathematical law, would you believe it?
In 1941, Soviet physicist Andrey Kolmogorov proposed the famous turbulence energy spectrum law: in turbulence, energy distribution over wavenumber follows a power law with exponent –5/3. More than eighty years later, a group of Chinese researchers, deep within the “brain” of large language models, measured the same number—5/3.
A question: Does the vector also hide “Zipf’s law”?
Languages have long exhibited famous statistical regularities, such as Zipf’s law—when words are ordered by frequency, frequency is inversely proportional to rank, and this law holds in almost every language. However, these patterns remain at the “word‑count” level. This paper asks: do the internal representations of large models—those vectors with hundreds or thousands of dimensions—also contain similarly simple, reproducible multi‑scale regularities?
a method: treat the article as a “flight path” in high-dimensional space
The researchers’ approach is very clever. They feed a passage of text into BERT or GPT, and each token—roughly understood as a fragment of a character or word—is mapped to a high‑dimensional vector. The whole passage becomes a trajectory in high‑dimensional space, like an aircraft’s flight path.
Instead of looking at the trajectory itself, they examine the “displacement” at each step: the difference between adjacent token vectors, called the “embedding step signal.” They then apply a Fourier transform to this signal—much like decomposing a piece of music into low, mid, and high frequencies to see how much power each band carries. Low frequencies correspond to meaning fluctuations spanning dozens or hundreds of words, while high frequencies capture fine‑grained, word‑by‑word changes.
For a fair comparison, all texts are uniformly truncated to 1,200 tokens; the corpus is also very diverse—Chinese includes academic papers, news, movie reviews, and even essays in the style of the Gaokao, while English covers scholarly articles, news, and movie reviews. The AI‑generated reference texts are produced in bulk by DeepSeek‑V3 on the same topics.
Result:
Four languages, humans and AI, all aligned on the same straight line
The results were surprising. For four languages—Chinese, English, German, Japanese—and two types of text—human‑written and AI‑generated—every power spectrum across a frequency range spanning roughly an order of magnitude forms a straight power‑law line, with a fitted exponent extremely close to 5/3 and an R² goodness‑of‑fit exceeding 0.99. In the algorithm’s “eyes,” human‑written articles and machine‑written articles exhibit the same multi‑scale rhythm.
Even more astonishing, the curves for human text and AI text almost perfectly overlap. And this is not an illusion supported by a few dimensions: over 90% of the embedding dimensions have exponents within ±10% of 5/3.
Figure 1: (a) Power spectrum of the embedded step signal under Chinese, English, German, and Japanese language/model configurations. The black dashed line is a 5/3 slope reference; the blue (AI‑generated text) and red (human text) curves almost completely overlap. (b) Polar heat map of the power spectrum by dimension: radial axis is frequency, angular axis is embedding dimension index, and similar spectral structures appear across the vast majority of dimensions.
Control experiment: not a by‑product of word frequency
Could this just be a by‑product of word‑frequency statistics? Two control experiments refute that point:
First, replace with context‑independent static word vectors (Word2Vec, GloVe), and the 5/3 scaling disappears. Second, randomize the token order before analysis, and the 5/3 also disappears. This shows that the multi‑scale organization comes from the joint weaving of context and word order, not from the words themselves.
The researchers also dissected the Transformer layer by layer: the shallow layers deviate exponentially from 5/3, and as you go deeper they approach it, with the deepest few layers yielding the cleanest 5/3. It seems that the model “digests” context layer by layer, converging toward this universal curve.
Figure 2: (a) Summary of the fitted exponent α for each language‑source configuration: contextual embeddings (blue/red circular markers) cluster around 5/3 (black dashed line), while static word vectors and the shuffled‑order control group deviate markedly. (b) Variation of the fitted exponent with Transformer layer number: the exponent overall approaches 5/3 as depth increases; the illustration shows a representative spectrum from an intermediate layer (α≈1.47).
A cautious analogy: Is it “turbulence”? Can’t draw an equals sign yet
The 5/3 exponent matches the Kolmogorov spectrum of turbulence, but the authors are very cautious, emphasizing that this is only a reference: no power law was derived from any equation, nor was a cross‑scale “energy flux” measured. Whether some information‑theoretic mechanism underlies this remains an open question.
Moreover, the German and Japanese corpora only cover the news domain, and other language types still need to be tested. The appendix also contains a detail that rewards careful readers: the longer the text, the wider the power‑law range—reminiscent of the turbulence phenomenon where “greater scale separation yields a broader inertial range,” yet the authors explicitly state that the two should not be equated.
Even so, this work already provides a new measuring stick: a quantitative fingerprint independent of any specific model, which can be used to compare multi‑scale structures across different languages, corpora, and model families, and may become a complementary benchmark for evaluating language models.
Language—could it also be in some kind of “turbulent” state? We don’t know the answer yet. But at least now we have a tool to measure it.
Causal Emergence Season 7 — From Theory to Practice
In the generation of consciousness in the nervous system, the evolution of urban traffic congestion, and the coordination and instability of global industrial systems, there is always a causal thread that runs from micro to macro: individual behavior may be simple, yet it can give rise to highly organized, irreducible overall structures across scales. Complex phenomena are not a linear superposition of microscopic rules, but rather emerge from causal organization formed gradually under multi‑scale dynamical interactions. It is against this backdrop that the theory of causal emergence was proposed, and advanced through works such as Causal Emergence 2.0, engineered emergence, and multi‑scale causal abstraction, gradually developing a research framework that integrates dynamical analysis, information‑theoretic metrics, spectral methods, and AI tools, thereby shifting the focus from “complexity itself” to “how causal structures arise, how they are measured, and how they function in real systems.”
To systematically review the latest advances in the field of causal emergence, Professor Zhang Jiang from the School of Systems Science at Beijing Normal University and founder of the Jizhi Club leads the initiative“Causal Emergence Season 7” reading group, organizing researchers and explorers interested in this topic to jointly study cutting‑edge literature and exchange research ideas. The reading group will be held online every Sunday starting February 22, 2026 (tentative time for the group: 10:00‑22:00), lasting about 10 weeks, and will include keynote presentations, discussions, and post‑session video recordings. Researchers in related fields and interdisciplinary enthusiasts are warmly invited to participate.
6. The gaokao score is just a ticket; your research adventure begins here!
7. Join the Jizhi Subtitle Team: Become a “Network Weaver” in the Complex Science Knowledge Community
Click “Read Original”, sign up for the book club
Feed URL: https://decemberpei.cyou/rssbox/wechat-jizhijvlebu.xml
Feed description:
WeChat Official Account – Jizhi Club
Feed icon:
Feed favicon: https://www.google.com/s2/favicons?domain=https%3A%2F%2Fwww.decemberpei.cyou%2Frssbox%2F
Collection task ID: 164
