Nums AI Releases Causilo Tabular Foundation Model

Nums AI has released Causilo, a pretrained tabular foundation model designed for classification and regression tasks. This model is available with a scikit-learn interface, its code released under an Apache-2.0 license, and pretrained weights accessible on Hugging Face. Causilo has demonstrated superior performance on the TabArena benchmark, achieving the highest Elo rating among single models for both classification and regression tasks. The model is currently deployable for research and evaluation purposes on both CUDA and CPU hardware. However, commercial, production, and hosted API usage requires a separate license agreement with Nums AI.

Causilo operates as an in-context learning model, meaning that the `fit` function does not alter the pretrained weights. Instead, it stores the training rows as context and predicts query rows through a single forward pass. According to its submission details for TabArena, Nums AI pretrained Causilo exclusively on synthetic data, without utilizing any of the TabArena datasets themselves. The model accepts inputs in the form of NumPy arrays or pandas DataFrames, with support for categorical features and missing values. For classification tasks, Causilo supports up to 10 distinct classes. By default, the regression output provides mean predictions. The latest version, 1.0.1, introduces median and quantile outputs, derived from 999 native quantiles.

The architecture of Causilo is divided into three distinct phases: refinement, compression, and in-context learning. The publicly available code and configuration files provide detailed insights into the operational mechanics of each phase. Features within the model are organized into sets of three, with each individual value being embedded using 16 learned sine and cosine frequencies. Missing values are handled by their own learned vector representation. Two column stages are employed to summarize each feature group. Within these stages, 128 latent slots are responsible for reading only the training rows and subsequently passing this summarized information to every row. Between these two column stages, a row stage facilitates interaction between feature groups through four latent tokens. This interaction is managed using cross-attention mechanisms rather than full self-attention, a design choice that Nums AI states maintains computational cost linear to the number of features. Following this, a pooling block compresses each row into a fixed 512-dimensional vector. Labels are appended to the training rows. Finally, a 12-layer prediction block enables query rows to attend to these labeled training rows, with the constraint that query rows cannot modify the training context or interact with each other.