Sakana AI Researchers Propose Layer-Local Training Method

Sakana AI researchers have introduced Augmented Lagrangian Predictive Coding (PC-ALM), a novel training method for deep neural networks that operates on a layer-local basis, offering an alternative to the traditional backpropagation algorithm. Backpropagation is characterized by a sequential process involving a forward pass, a backward pass, and a weight update, where each step is dependent on the completion of the preceding one. This global, phase-locked mechanism lacks a clear biological parallel in brain function, prompting ongoing research into local-learning alternatives like predictive coding (PC). PC-ALM aims to address limitations in existing PC methods by enabling layer-local updates while still achieving credit signals that align with backpropagation's performance. The research team reported successful training of residual MLPs with up to 1000 layers, achieving results within approximately 2 percentage points of backpropagation accuracy on the MNIST dataset. This new method is presented as a training technique rather than a distinct model, and an MIT-licensed JAX reference implementation is available, capable of running on CPUs and reproducing the paper's experimental findings across various network widths and depths. The research has, however, been limited to testing on smaller image benchmarks. Standard predictive coding methods often encounter difficulties when training deep, narrow networks. In these networks, PC treats each hidden activation as an optimization variable and penalizes the discrepancy between the layer's activation and the prediction received from the layer below. While inference in PC involves gradient descent on this energy function and learning uses a Hebbian-like weight update, the diffusion of supervision from the output layer to the input layer can lead to a fading credit signal in deep, narrow architectures. This issue, characterized by Innocenti et al. as a PC-BP gap dependent on network width and depth, is most pronounced when the width is significantly smaller than the depth. PC-ALM modifies the training objective by framing it as minimizing the supervised loss subject to layer-wise constraints. While standard PC employs a quadratic penalty relaxation of these constraints, PC-ALM utilizes the augmented Lagrangian method. This approach incorporates a Lagrange multiplier, denoted as \(\lambda_i\), for each layer's constraint, maintaining the predictive coding penalty while introducing a more robust mechanism for propagating error signals through the network. The researchers' work aims to bridge the performance gap between biologically plausible local learning rules and the highly effective, albeit less biologically realistic, backpropagation algorithm.