Yifan Zhang Unveils Recurrent Looped Transformer

AI researcher Yifan Zhang has released the code, paper, and project page for the Recurrent Looped Transformer (RLT) under an Apache 2.0 license. RLT introduces a novel causal encoder plus recurrent decoder architecture designed to carry the decoder's hidden state and sliding-window KV cache across every prompt and response token. Unlike conventional causal Transformers that route tokens through a fixed stack of layers, RLT extends its causal computation path as the sequence grows, keeping the decoder's layer count fixed.

In RLT, with a decoder depth of L_D, a token at position t exists at the end of a recurrent path containing t multiplied by L_D decoder-block applications. The reference configuration features 48 encoder layers and 48 decoder layers, sharing attention and feed-forward network weights. Each decoder block also executes cross-attention over the encoder memory, which increases its floating-point operation count beyond that of a standard encoder block. This setup unifies prefill, decode, pretraining, supervised fine-tuning, and current-policy reinforcement learning under a single state transition.

While the architecture aims to scale test-time compute and reasoning, early empirical results are mixed. In independent 79,000-parameter state-tracking tests, a plain Gated Recurrent Unit (GRU) outperformed RLT at four times the training length. Furthermore, the technical report lacks measured evidence that RLT's temporal path actually improves reasoning. For practitioners, RLT offers an experimental framework to explore recurrent state-carrying mechanisms, but it currently lacks the proven performance advantages of established architectures.