Continuous Thought Machines: The Future of AI Reasoning?

Every transformer model you've used shares the same dirty secret.

It doesn't think in time. It thinks in layers.

Feed GPT-4, Claude, or Llama a maze, and the model doesn't "look" at it step by step the way you would with a pencil. It processes the whole thing in one parallel pass through a fixed stack of layers, then spits out an answer. There's no internal clock, no sense of "still working on this."

That's the bottleneck Sakana AI is going after with the Continuous Thought Machine, or CTM. I spent the better part of two weeks running their open-source implementation, poking at the interactive maze demo, and reading the 60-page paper closely enough to argue with it. Here's the honest version of what I found.

The Core Problem CTMs Are Trying to Solve

Modern deep learning made a trade decades ago: drop biological realism, keep the math simple.

A standard artificial neuron takes inputs, multiplies them by weights, sums them, and fires through an activation function. Once. No memory of its own past activity.

Real neurons don't work that way.

• Biological neurons fire at different rates depending on recent history.
• Timing between spikes carries information, not just the spike itself.
• Groups of neurons synchronize to represent concepts, not just individual cells lighting up.

Transformers approximate intelligence by brute-forcing scale on top of that simplified neuron. It's worked spectacularly well. It's also why models still faceplant on tasks that require genuine step-by-step deliberation rather than pattern completion.

Sakana's bet: bring the time dimension back into the neuron itself.

What a Continuous Thought Machine Actually Is

Strip away the branding and CTM comes down to two mechanical changes to how a neural network processes information.

1. Neuron-level temporal processing

Every neuron in a CTM keeps a short history of the signals it has received. Instead of one weight per input, each neuron applies its own tiny learned function across that history to decide when and how strongly to fire.

2. Neural synchronization as the actual output representation

This is the part that took me a re-read to fully absorb. In a CTM, the model doesn't read its answer off a final layer's activations. It reads the answer off how synchronized pairs of neurons become with each other over internal "thinking ticks."

Pro Tip: If you're trying to build intuition for synchronization matrices before touching the code, run the browser-based maze demo on Sakana's site first. Watching the attention trace crawl through the maze in real time made the synchronization concept click for me faster than three passes through the paper's math.

The model runs for a variable number of internal steps — its own private "thinking dimension" — completely decoupled from how much actual input data it's given. A static image and a video clip can both be reasoned over using the same internal clock.

How This Differs From Transformers and RNNs, Practically

I keep seeing CTMs described online as "just RNNs with extra steps." That undersells what's different, and overselling isn't right either. Here's the honest breakdown.

Architecture	Time handling	Output source	Compute pattern	Best-known strength
Transformer	None natively (positional encodings simulate order)	Final layer activations	Fixed-depth parallel pass	Massive parallel scale, language fluency
RNN / LSTM	Sequential, one step per token	Final hidden state	Sequential, one pass per input step	Streaming sequence data
Continuous Thought Machine	Internal thinking ticks, decoupled from input length	Neural synchronization over time	Variable internal iterations per input	Step-by-step deliberation, interpretability

The row that matters most for developers: compute pattern. A transformer's cost scales with input length. A CTM's reasoning depth is a knob you can turn independently of the input. That's the part that has robotics and planning researchers paying attention.

Under the Hood: What Happens When a CTM "Thinks"

Here's the sequence I traced through the reference implementation while debugging a broken checkpoint load (more on that disaster below).

1. Input encoding — Raw data (image patches, tokens, sensor readings) gets embedded, same as most architectures.
2. Recurrent internal ticks begin — The model doesn't move to an output layer yet. It starts iterating internally, each neuron updating its private history buffer.
3. Synchronization matrix builds — At each tick, pairwise synchronization values between neurons get computed and accumulated.
4. Certainty-based stopping — The model can effectively decide it's "thought enough" based on how stable its synchronization pattern has become, rather than a hardcoded number of layers.
5. Output decoded from synchronization, not activation — The final prediction is read from the synchronization representation, which is where CTM diverges hardest from everything else on the market.

That fourth step is the one I find genuinely novel rather than dressed-up. Most architectures fire for a fixed compute budget regardless of task difficulty. A CTM, in principle, can spend longer on a hard maze and less on an easy one.

In my testing, that adaptive behavior was visible but not dramatic on the small demo models — you're seeing tens of ticks difference, not orders of magnitude. Sakana is upfront that this is early-stage research, not a shipped product.

Benchmark Reality Check

I want to be blunt here because a lot of coverage glossed over this.

On ImageNet-1K, the CTM in the paper hit 72.47% top-1 accuracy. Modern ConvNeXt and ViT variants clear that by a wide margin.

That is not a knock on the architecture. Sakana didn't optimize CTM for leaderboard chasing — they optimized for behaviors transformers structurally can't produce, like solving 2D mazes from raw pixels with zero positional embeddings, purely through the sequential internal reasoning process.

Where CTM actually pulled ahead in ways that mattered to me:

• Maze navigation without spatial priors — the model builds its own implicit sense of position through synchronization dynamics, which transformers need explicit coordinate embeddings to fake.
• Calibrated uncertainty — the model's confidence tracked task difficulty more closely than a same-size transformer baseline in the paper's comparisons.
• Interpretability — you can actually watch attention move across an image tick by tick, which is a genuinely useful debugging window transformers don't hand you for free.

Pro Tip: When benchmarking CTM checkpoints locally, log synchronization matrix size before you scale up neuron count. It grows faster than people expect, and I torched 40 minutes of compute time before realizing my "efficient small model" was quietly allocating a synchronization tensor bigger than the model weights themselves.

Setting It Up Yourself: What Actually Trips People Up

If you want to run this rather than just read about it, here's the practical path, based on what actually broke for me versus what the README implies will go smoothly.

• Clone the official SakanaAI repository rather than any mirror — early forks had stale checkpoint loading code that silently produced garbage synchronization outputs instead of erroring out.
• Start with the maze or CIFAR-10 examples before touching ImageNet-scale training. The internal tick loop makes training meaningfully slower per epoch than an equivalent transformer, and you want to feel that cost on a small task first.
• Watch your GPU memory closely once you go past toy neuron counts — the synchronization representation is where memory pressure sneaks up, not the embedding layers like you'd expect from transformer habits.
• Read the "Discussion and future work" section of the paper before the method section. It sets honest expectations about where this is genuinely early research versus production-ready.

Pro Tip: If training seems to be converging unusually slowly compared to your transformer intuition, don't assume your learning rate is wrong first. The internal tick count is doing double duty as both a compute budget and an implicit regularizer — tune that before you touch the optimizer.

The Honest Limitations

Nobody should walk away thinking CTM is a drop-in transformer replacement. It isn't, and Sakana doesn't claim it is.

• Training cost is higher per sample. The internal tick loop means more sequential compute per input than a comparably sized transformer's single parallel pass.
• Tooling is immature. There's no equivalent of the transformer ecosystem — no mature quantization tricks, no widely tested serving frameworks, no fine-tuning recipes battle-tested across a thousand GitHub repos yet.
• Scale is unproven. Everything published so far tops out well below frontier LLM parameter counts. Whether synchronization-based reasoning holds up at hundred-billion-parameter scale is genuinely unknown.
• Benchmark parity isn't there yet on raw classification tasks. If your use case is "beat ResNet at ImageNet," this isn't your architecture today.
• Interpretability is a double-edged sword. Watching synchronization patterns is fascinating for researchers; it's not yet packaged into a form a product team could use to debug a customer-facing failure.

FAQ

Is a Continuous Thought Machine the same thing as chain-of-thought prompting? No, and this confuses people constantly. Chain-of-thought is a prompting technique applied to transformers that makes them generate intermediate reasoning text. A CTM builds "thinking" into the model's internal architecture at the neuron level, independent of any text output.

Can I use CTMs for language modeling today? Not practically. Current public work focuses on vision, mazes, and reinforcement learning tasks. Nobody has published a competitive CTM-based language model at meaningful scale yet.

Does CTM require special hardware? No exotic hardware requirements — it runs on standard GPUs. The catch is compute efficiency, not compute type: the sequential internal ticks mean you'll want more VRAM headroom and patience than an equivalently sized transformer.

Is Sakana AI planning to scale this up further? The paper's own discussion section frames CTM as a research direction rather than a finished product, and points toward scaling and broader task generalization as open next steps. Nothing officially announced beyond that as of this writing.

Where This Actually Goes

CTMs aren't going to dethrone transformers next quarter, and anyone telling you otherwise is selling something.

What they represent is a legitimate crack in the assumption that scale-plus-attention is the only road to better reasoning. The synchronization idea is weird enough, and grounded enough in actual neuroscience, that I'd bet on some version of "timing as representation" showing up inside hybrid architectures within the next couple of research cycles — even if pure CTMs never replace transformers outright.

If you're a developer who wants to be ahead of that curve rather than reading about it after the fact, the SakanaAI GitHub repo and the interactive maze demo are worth an evening of your time. Not because you'll ship a product with it next week. Because watching a model visibly deliberate, tick by tick, changes how you think about what "reasoning" in a neural network could even mean.