Perplexity Open Sources Lily Inference Engine for Apple Silicon

Perplexity has open-sourced Lily, a local inference engine that powers its Hybrid Compute feature, enabling efficient large language model execution on Apple Silicon hardware. This single-process runtime is built using Rust for its core logic and Metal, Apple's graphics and compute API, for model execution, deliberately omitting PyTorch and MLX from the critical path. Lily is specifically designed for a single model, Qwen3.6-35B-A3B, and a singular hardware family, Apple Silicon, with its narrow focus being the key to its performance advantages. The engine provides an OpenAI-compatible chat-completions API that streams tokens and is available as a standalone demo within the pplx-garden repository. The 4-bit quantized checkpoint for the Qwen3.6-35B-A3B model is 19.4 GB, necessitating an Apple Silicon Mac with a minimum of 32 GB of unified memory for realistic operation. Perplexity's commercial Hybrid Compute product recommends macOS 15 or later, with 24 GB of RAM as the minimum requirement and 32 GB for optimal performance. The decision to specialize Lily stems from the desire to achieve performance gains beyond the capabilities of the default Mac AI stack, which typically utilizes MLX and MLX-LM. While the MLX stack already includes a Qwen implementation with grouped expert routing, fused Metal kernels, and Grouped-Query Attention (GQA), its operations must maintain cross-architecture reusability. Lily, conversely, sacrifices this generality to deeply integrate model structure, execution plans, and kernel selection within a single, optimized runtime. The Qwen3.6-35B-A3B model itself is a significant component, boasting 35 billion parameters and activating approximately 3 billion parameters per token. Its architecture features a router that scores 256 experts and selects eight, in addition to a shared expert that processes every token. The model also incorporates 10 full-attention layers utilizing grouped-query attention (with 16 query heads and two key-value heads) alongside 30 Gated DeltaNet layers. This complex structure results in three distinct processing patterns: uneven expert group activation, attention mechanisms that operate over a growing key-value cache, and a fixed-size recurrence mechanism. During the prefill phase, Lily prioritizes keeping weights packed and routing operations on the GPU. The model checkpoint employs groupwise affine 4-bit quantization, where each group of 64 weights shares a bfloat16 scale and bias, compressing the original approximately 70 GB of bfloat16 weights down to 19.4 GB. Metal tensor operations utilize bfloat16, requiring the weights to be in this format for efficient processing.