The question of how transformer architectures can perform iterative reasoning remains fundamentally underexplored. While large language models demonstrate impressive few-shot capabilities, their ability to perform genuine multi-step reasoning—particularly on constrained combinatorial problems—remains opaque. This paper tackles a deceptively simple question: can a single transformer block, equipped with adaptive computation mechanisms, solve structured reasoning tasks if given access to learned memory? The answer, surprisingly, is no—not without careful architectural and initialization choices. This work exposes critical vulnerabilities in how we train adaptive computation systems and provides empirical evidence that memory tokens function as essential computational scratchpads rather than optional optimizations.

Understanding these constraints matters because Adaptive Computation Time (ACT) and Universal Transformers represent promising directions for making transformer inference more efficient. If we can't reliably train these systems on well-defined benchmarks, scaling them to real-world reasoning tasks becomes substantially harder. The paper's identification of a widespread initialization trap—one that silently fails in over 70% of training runs—suggests that previous negative results on similar architectures may reflect implementation artifacts rather than fundamental limitations.

The experimental setup is elegantly constrained: researchers evaluate a single-block Universal Transformer on Sudoku-Extreme, a combinatorial reasoning benchmark requiring constraint satisfaction over 81 cells. The architecture operates in a recurrent loop, where each step applies the same transformer block to accumulated hidden states. Adaptive Computation Time allows the model to dynamically determine how many recursive steps to execute before producing output, controlled by a learned halting mechanism. The critical variable under investigation is the number of learned memory tokens T available as a computational scratchpad—separate from input tokens.

The empirical findings reveal striking non-linearity. With T=0 (no memory), performance is essentially zero across all tested configurations. T=4 produces borderline results, while T=8 reliably achieves 57.4% exact-match accuracy on 81-cell puzzles. Performance plateaus between T=8 and T=32 (57.4% ± 0.7%), then collapses at T=64 due to attention dilution—a phenomenon where excessive tokens fragment attention patterns, reducing effective information flow. This sharp threshold behavior suggests memory tokens serve a qualitatively different function than standard input embeddings; they appear to enable a specific computational mode rather than providing marginal capacity improvements.

The most significant contribution may be identifying the "router initialization trap," a failure mode in ACT training that has likely gone undiagnosed in prior work. The halting mechanism in ACT uses a learned router that produces a probability of termination at each step. Standard initialization schemes—zero-bias (yielding ~50% halt probability) and Graves' recommended positive bias (~73% halt probability)—both cause the model to halt prematurely during early training, typically after 2-3 steps. The model then settles into a shallow equilibrium where it consistently halts around step 5-7, and crucially, cannot escape this attractor even with continued training. This is not a convergence to a locally optimal solution; it's a training failure mode. The authors demonstrate that inverting the bias to -3 (producing ~5% initial halt probability) eliminates this trap, allowing the model to explore deeper computation paths. Ablation studies confirm this failure is inherent to ACT's initialization dynamics, not an artifact of their specific architectural choices.

With reliable training established through proper initialization, the authors conduct systematic comparisons. ACT processing yields more stable results (56.9% ± 0.7% across 3 seeds) compared to fixed-depth variants (53.4% ± 9.3%), demonstrating that adaptive computation reduces variance. Introducing lambda warmup—gradually increasing the penalty on computation steps during early training—achieves matching accuracy (57.0% ± 1.1%) while reducing average ponder steps by 34%, suggesting the model initially explores unnecessary depth before learning efficient computation patterns.

Analysis of learned attention patterns reveals functional specialization across recursive depths. Memory-reading heads focus on retrieving information from the scratchpad tokens, constraint-propagation heads enforce Sudoku rules by propagating information across the grid, and integrator heads synthesize information to produce final predictions. This specialization emerges without explicit supervision, suggesting that multi-step reasoning naturally decomposes into these cognitive operations.

CuraFeed Take: This work is deceptively important for practitioners implementing adaptive computation systems. The initialization trap alone represents a major reproducibility hazard—researchers publishing negative results on Universal Transformers may have unknowingly fallen into this failure mode, leading to premature dismissal of promising architectures. The sharp memory threshold (8 tokens mandatory, 64+ tokens counterproductive) suggests that memory-augmented reasoning doesn't scale smoothly; there appear to be discrete computational regimes where different amounts of working memory enable qualitatively different solution strategies. The most actionable insight is that ACT initialization deserves as much attention as learning rate scheduling—treating it as a hyperparameter to tune rather than a fixed constant could substantially improve reproducibility across the broader literature. For researchers designing reasoning-capable transformers, this implies that explicit memory mechanisms aren't luxuries but prerequisites, and that the interaction between memory capacity, depth, and attention patterns requires careful empirical characterization for each task domain. Watch for follow-up work applying these insights to larger-scale reasoning problems and exploring whether the initialization trap generalizes to other adaptive computation mechanisms.