Sakana AI Trains Networks Without Backpropagation

Sakana AI has developed a novel method for training neural networks that bypasses the need for backpropagation, a cornerstone of deep learning that faces challenges with biological plausibility due to its reliance on exact weight matrix transposes. The research, detailed in their paper "Diffusing Blame," introduces an Error Diffusion (ED) mechanism that enables networks to learn while adhering to Dale's principle, which states that a neuron is either exclusively excitatory or exclusively inhibitory. This approach avoids the "weight transport problem" inherent in backpropagation.

Error Diffusion, first proposed by Kaneko in 2000, is a local learning rule where each weight update is determined by only three signals: presynaptic activity, a postsynaptic activation derivative, and a single global error sign. This locality makes ED compatible with Dale's principle. Previous applications of ED were limited to binary classification and the MNIST dataset. Sakana AI's advancement lies in extending ED's capabilities to more complex tasks and datasets.

To accommodate Dale's principle and enable ED, Sakana AI implemented a dual-stream architecture for each layer. This design splits each layer into an excitatory stream (p) and an inhibitory stream (n). The forward pass calculates preactivations for each stream by combining inputs from both streams with specific weight matrices. Crucially, all four weight sub-matrices within this dual-stream structure remain non-negative element-wise, ensuring that learned weights are either purely excitatory or inhibitory. This architecture, while increasing parameter count by approximately four times compared to single-stream networks, allows for the application of Error Diffusion.

With the dual-stream architecture in place, the key innovation is "modulo error routing." This technique extends Error Diffusion beyond binary classification tasks. The research team's implementation achieved notable performance metrics, reaching 96.7% accuracy on the MNIST dataset and 61.7% accuracy on the CIFAR-10 dataset. These results demonstrate the potential of Error Diffusion and dual-stream networks as a viable alternative to backpropagation for training deep learning models, particularly in scenarios where biological plausibility or avoidance of the weight transport problem is a priority.