Princeton Researcher Proposes Recurrent Looped Transformer

A technical report authored by Princeton researcher Yifan Zhang introduces the Recurrent Looped Transformer (RLT), a novel architecture for decoder-only Large Language Models (LLMs) designed to enhance temporal depth. Unlike conventional LLMs where information flow between consecutive tokens is primarily mediated by attention mechanisms over cached keys and values, RLT proposes a continuous loop. Specifically, the decoder's final hidden state and its layerwise sliding-window attention (SWA) cache are systematically passed to the subsequent token's processing, without any reset at token boundaries. This mechanism applies across both the prompt and the generated response sequences.

The RLT architecture is presented as a design specification, detailing its architectural components, execution schedules, and a proposed reinforcement learning (RL) replay contract. The report explicitly states that it does not include any measured results concerning efficiency, reasoning quality, or scaling performance. The RLT design integrates a causal encoder with a recurrent decoder. The encoder processes tokens in parallel, adhering to a causal mask, and generates representations denoted as e_t. From these representations, a key-value memory M≤t is projected. This memory can be shared across all decoder layers (G = 1) or maintained independently for each layer (G = L_D).

The core of the recurrence lies within the decoder. The complete state of the decoder at time t is represented as H t = (s t , C t D ), where s t signifies the final decoder output and C t D encapsulates the retained SWA keys and values across every decoder layer. For each token processed, a gated merge operation combines the encoder representation e_t with the output from the preceding token, s {t-1}. Subsequently, each decoder block executes causal SWA over its own activations, performs cross-attention to the encoder memory, and applies a feed-forward network (FFN). The sliding window W is designed to include the current token, ensuring that a maximum of W – 1 historical entries per layer are preserved. The distribution for the next token is derived from s t . The initialization process occurs only once before the beginning-of-sequence (BOS) token, utilizing a learned start state s* and an empty cache.

A reference configuration described in the report employs 48 encoder layers and 48 decoder layers. Crucially, the attention and FFN weights are shared between the encoder and decoder layers, indicating a form of parameter reuse rather than activation copying. This configuration means that for each token, a total of 96 logical blocks are executed. While the decoder blocks incorporate cross-attention, leading to unequal FLOPs per block compared to encoder blocks, the fundamental principle is the continuous flow of state. Zhang refers to this as parameter reuse. The design is underpinned by three core design principles, though these are not elaborated upon in the provided text.