Can Thermodynamics Reveal What Language Models Learn About Time and Cause?

Reverse two sentences, and an AI model may reveal what it learned

“The glass fell to the floor. It broke.” The sequence makes sense because the second event follows the first. Reverse the sentences—“The glass broke. It fell to the floor”—and the same words describe a less natural sequence. People have background knowledge about gravity, impacts and fragile objects. Does a language model capture anything comparable, or merely repeat familiar patterns of text?

A new paper from the University of Tokyo offers an unexpected tool for asking that question. Takahiro Sagawa, a professor of applied physics, has developed a stochastic-thermodynamic framework for analyzing the directionality of sequences produced by generative models. Published in the peer-reviewed journal PRX Intelligence on November 5, 2026, and announced by the university and Japan Science and Technology Agency on November 6, the work probes AI behavior without claiming to explain humanlike intelligence.

A thermodynamic result that does not measure electricity or heat

The pairing of AI and thermodynamics invites a misleading interpretation. The study does not calculate the watts consumed by a graphics processor, estimate data-center cooling demand or show that a language model can be made more energy-efficient. Its central quantity, “entropy production,” measures a statistical asymmetry between a sequence and its time-reversed counterpart.

In ordinary physical thermodynamics, entropy production is connected to irreversible processes and energy dissipation. Here the mathematical language is applied to the probabilities of generated strings and other time series. The University of Tokyo expressly warns that this is not the physical heat or energy use of the hardware running the model. That boundary is essential to understanding the result.

From steam engines to information theory

The origins of thermodynamics lie in efforts to understand heat, work and the limitations of engines. Nineteenth-century physics made irreversibility central: some transformations occur naturally in one direction but not the other. Statistical mechanics later tied those macroscopic observations to the probabilities of many microscopic configurations. Information theory developed a different, though mathematically related, notion of entropy to describe uncertainty.

The connection between measurement, information and physical work has been a major subject of modern research. In papers from 2008 and 2010, Sagawa and Masahito Ueda studied thermodynamic laws involving feedback control and information gained by measurement. The new paper extends a long-running program of asking how information and irreversibility interact, this time through the outputs of computational models rather than a conventional heat engine.

What makes language generation autoregressive

A language model often constructs a response one token at a time. At each step, it uses the preceding context to assign probabilities to possible next tokens, samples or selects an output, and repeats. A token can be a whole word, a fragment of a word or another unit used by the model; treating all tokens as complete words is inaccurate.

The category of autoregressive generative models extends well beyond chatbots. Recurrent neural networks, Transformer models, state-space architectures and certain representations of Kalman filters share a structure in which a summary of past outputs informs the next output. The influential 2017 Transformer paper, Attention Is All You Need, helped reshape language-model development, but Sagawa’s framework is not limited to Transformers.

The hidden challenge of memory

For an elementary Markov process, the next observation can be described from the present observed state. Text seldom behaves so simply. The probability of a pronoun, a conclusion or the next action in a story may depend on sentences far earlier in the passage. The observed token sequence is generally non-Markovian: knowing the previous token alone is not enough.

Inferring forward and reverse probabilities directly from every possible observed history quickly becomes prohibitive. Sagawa exploits a shared architectural feature: an internal latent state summarizes the history before the next output is sampled. This makes it possible to estimate entropy production from generated trajectories without enumerating exponentially many possible histories. The mathematical achievement is as important as the language-model demonstration.

Why reversing individual tokens gives the wrong impression

Suppose a study reversed every token in a well-formed paragraph. Most reversed paragraphs would be syntactically strange, even where they carried no meaningful causal story. A model assigning them lower likelihood might simply be recognizing grammatical conventions. That says little about its representation of how events unfold in the world.

The new work therefore distinguishes token-level from sentence-level entropy production. For the latter, the order of sentences is reversed while the words within each sentence remain in their original order. The contrast is designed to reduce a superficial grammatical confound and make temporal ordering across statements easier to study. It does not remove every possible linguistic bias.

What GPT-2 and Qwen3-4B-Base actually showed

The proof-of-concept analysis used the pretrained GPT-2 model with 124 million parameters and was reevaluated with Qwen3-4B-Base, a substantially larger open-weight model. The researchers compared sets of sentences describing causally ordered events with sets presenting comparatively independent facts.

Sentence-level entropy production tended to be greater in the ordered-event texts. In other words, the model assigned a stronger relative asymmetry to generating the coherent sequence rather than its reverse. The result suggests that aspects of event ordering are reflected in generated-text probabilities. It is not a demonstration that either model can reliably infer causation, reason about physics in unfamiliar settings or maintain a complete representation of reality.

The world-model debate becomes measurable—but not settled

“World model” is a loaded term. In AI research it can mean an internal representation of relationships, entities and changes in the environment; it does not imply subjective awareness. Language models might acquire representations of spatial relations and temporal patterns by learning from text, but how far those representations extend is contested.

The new method supplies one quantitative probe of sequence directionality. It may help researchers ask whether a model treats causally connected sentences differently from unrelated statements. It does not identify every concept encoded in model weights or establish that the system understands events in the way people do. The university appropriately frames world-model analysis as a future research direction, not an accomplished diagnosis of AI understanding.

A surprising connection to hindsight and compression

Sagawa goes beyond providing a scalar score of irreversibility. The paper shows that total entropy production can be decomposed exactly into non-negative contributions from individual steps. The analysis uses retrospective inference: what can be inferred about an earlier event when later outputs are known?

These contributions can be interpreted in terms of compression loss and model mismatch. That structure offers a principled way of locating sources of temporal asymmetry within a generation process, rather than treating a completed sequence as an indivisible object. The author also derives an analytical expression in a linear Gaussian setting connected to the Kalman innovation representation, illustrating the framework’s breadth beyond neural text models.

What the researchers have not established

The university release identifies an important limitation: the evaluated text sets were created by prompting a commercial generative AI system. Those texts were categorized for experimental comparison, but the categories were not derived from a complete formal theory of causation. Writing style, length, topic selection and dataset construction can all matter when comparing sequence probabilities.

It remains to be tested how robust the observed effect is on independently curated benchmarks, multilingual corpora, different model families and settings that require reasoning beyond familiar narratives. A suggestive statistical separation is not equivalent to a validated test of causal understanding. Nor can results from two tested open-weight models automatically be generalized to all commercial language models.

Why researchers in physics and AI should care

Many evaluations of generative AI focus on whether a model produces correct answers, follows instructions or performs well on a benchmark. Those are important measures, but they do not directly disclose the probability structure behind the model’s outputs. A thermodynamic perspective offers a complementary line of inquiry: how sharply does a system distinguish the forward and reversed versions of a process?

For physicists, the study carries the tools of nonequilibrium thermodynamics into a broad class of artificial sequence generators. For AI researchers, it suggests a measurement strategy for temporal structure. For industry, it is early-stage science, not an engineering guarantee about more accurate chatbots, faster processors or lower power bills. A reliable practical application would require much more comparative validation.

A glass, a sequence, and the next question

One can reverse two sentences in seconds. Understanding why a generative model prefers one order requires defining the process, estimating probabilities and separating grammar from the structure of events. The new research turns that seemingly trivial reversal into a carefully posed mathematical question.

From steam-era inquiries into irreversibility to the mathematics of information and modern neural networks, the same intellectual thread keeps returning: what does directionality reveal about a system? Sagawa’s paper does not prove that a machine possesses a mental model of the world. It provides a rigorous new instrument for asking how much of a world’s temporal order may be reflected in the texts machines generate.