Can Thermodynamics Reveal How AI Understands Time? A New Theory from Tokyo

A broken cup, two sentences, and a question about time

A glass falls from a table. It shatters on the floor. Reverse the order of those statements and something feels wrong, even when the sentences themselves remain grammatical. We recognize a sequence of events: one precedes, and helps explain, the other. A language model may favor the same ordering, but what would it mean to measure that preference rigorously?

A new paper from University of Tokyo physicist Takahiro Sagawa offers an answer rooted in stochastic thermodynamics. Published in the peer-reviewed journal PRX Intelligence on November 5, 2026, and announced in Japanese by the university and the Japan Science and Technology Agency the next day, the work develops a mathematical measure of irreversibility in autoregressive generative models. The result is a way to ask sharper questions about a model’s output—not proof that it experiences time or thinks causally like a person.

Why a language model and a Kalman filter belong in the same story

The common thread is sequential prediction. An autoregressive generator uses a summary of its previous outputs to assign probabilities to the next output. In a language model those outputs are tokens. In older engineering systems, a Kalman filter uses information accumulated over observations to estimate a changing state; under an appropriate generative interpretation, it fits the same broad mathematical pattern.

Sagawa’s framework covers Transformers, recurrent neural networks, state-space models including Mamba, and classical Kalman filtering. What links them is an internal state computed from prior history and a rule that converts that state into a probability distribution over the next observation. This is a more ambitious contribution than drawing a loose analogy between artificial intelligence and the second law of thermodynamics.

A century-old question: why does the order of events matter?

Classical thermodynamics emerged from problems of heat, work and efficiency in the nineteenth century. In later statistical and stochastic descriptions, irreversibility could be expressed by comparing the probability of trajectories followed in the ordinary direction of time with the probability of related trajectories run backward. A process that strongly prefers one direction has a measurable time asymmetry.

That comparison is central to the new research, but a crucial qualification matters. Entropy production here describes probabilities of symbol or signal sequences generated by mathematical models. It is not a measurement of the electrical energy a language model consumes, nor the heat emitted by its processors. The theory does not establish a way to reduce an AI data center’s power bill.

The obstacle of memory

Simple Markov processes can be described through the immediate present: once the current state is known, earlier history adds nothing to the next-step probability. The observable output of an autoregressive language model behaves differently. The next word can depend on a promise made several paragraphs earlier, the identity of a character, or a condition stated at the beginning of a document.

This dependence on extended history makes the observed sequence non-Markovian. A brute-force calculation that enumerates every possible history quickly becomes impossible. Sagawa’s key mathematical move is to exploit the deterministic way these models summarize history internally. Forward and backward likelihoods can be evaluated from generated sequences without the exponential explosion of enumerating all possible pasts. The paper also describes a Monte Carlo estimation procedure built on familiar log-probability computations.

Why reversing words would be a misleading test

The first obstacle in testing language models is embarrassingly simple: if words are reversed one by one, nearly any sentence turns into grammatical nonsense. A model might reject the result for purely syntactic reasons. That tells us little about its grasp of events. The paper explicitly reports that token-level entropy production is dominated by such a syntactic artifact.

The researchers therefore introduced a coarser test: preserve the words within each sentence while reversing the order of sentences. A sentence stating that a cup broke can then be placed before the sentence stating that it fell. This design does not eliminate every confounder, but it gets closer to comparing the order of described events rather than measuring the damage caused by scrambled grammar.

What the GPT-2 and Qwen tests actually found

Sagawa evaluated the idea with GPT-2 and conducted a further assessment with the larger Qwen3-4B-Base model. Across sets of texts, narratives with connected sequences of events tended to show larger sentence-level entropy production than collections of unrelated facts. In the paper, the GPT-2 differences reached statistical significance for three of five corpora; the Qwen assessment found significant differences across all five.

That is suggestive evidence that sequential language statistics carry information about temporal organization. It is not a controlled demonstration of causal reasoning. The test corpora were assembled using commercial generative AI prompted to produce the text. Wording conventions or other cues may contribute to the differences. The researchers themselves call for better-defined temporal and causal datasets.

A new probe, not a certificate of understanding

The phrase world model has become prominent in discussions of artificial intelligence. It refers to internal representations that may capture relationships among objects, places, events and changes over time. Researchers debate what language models learn about these structures from text, and how directly those structures correspond to the physical world.

Irreversibility offers one candidate probe. If a model assigns systematically different probabilities to plausible and implausible temporal orders, the gap might reveal something about what patterns it has acquired. Yet a successful score is not evidence of consciousness, humanlike comprehension or reliable causal reasoning. A model could exploit clues without representing the underlying mechanism in a robust way. The method earns its value through careful validation, not through sweeping claims about machine minds.

The surprising mathematics of looking backward

Beyond the language-model demonstration, the paper establishes an exact information-theoretic decomposition. Total entropy production can be represented as a sum of nonnegative contributions associated with individual steps. Each can be interpreted through a retrospective task: how well could a backward model infer an earlier observation from what came afterward?

Those contributions separate into two ingredients. Compression loss arises when a backward latent representation discards relevant details from the future. Model mismatch arises when a prediction rule developed for the forward direction is applied in reverse. This offers a potentially useful diagnostic vocabulary for understanding where a sequence’s directional asymmetry comes from, rather than reporting only a single aggregate number.

From nineteenth-century heat engines to twentieth-century control systems

The intellectual lineage is unusually broad. Nineteenth-century thermodynamics introduced the irreversibility problem. Twentieth-century statistical physics and information theory made probabilistic accounts more precise. Rudolf Kalman’s 1960 work on filtering became foundational to navigation, signal processing and control engineering, long before today’s large language models existed.

The Transformer architecture, introduced in 2017, enabled new methods for using extensive context in sequence prediction. Sagawa’s work meets these traditions at the level of probability. In the linear-Gaussian case associated with the Kalman innovation representation, the paper derives entropy production analytically and checks it numerically. That tractable example is important: it shows that the mathematical framework is not limited to opaque neural networks.

The unanswered questions that matter most

There is no claim here that researchers proved how language models understand time. Nor is there evidence that the new measure predicts factual accuracy, prevents hallucinations or guarantees safer AI. The published results concern one theoretical framework and a first set of experiments, not a finished benchmark for commercial systems.

Next steps include independent text datasets, comparisons across languages and model families, and tasks that separate mere statistical sequence cues from genuine causal generalization. The distinction is central to trustworthy science. A model can predict that the sentence about a broken glass follows the sentence about a fall. Finding out precisely why it does so—and when that apparent knowledge fails—is a more demanding question. This Tokyo research gives investigators a new way to pursue it.