Perplexity Details GPU Embedding Stack: Ivy, Tulip, ROSE

Perplexity's engineering team has detailed its under-the-hood infrastructure for serving its embedding models, specifically the pplx-embed models used across Perplexity Search, Computer, and its API Platform. The company published "Fast Embeddings on GPUs," an account focusing on the efficiency and cost-effectiveness of running these models on GPUs. Perplexity states that embedding inference on the GPU side has largely converged across engines, particularly on mature hardware like NVIDIA's Hopper and Blackwell architectures. The primary areas for improvement and competitive advantage lie not in the core models themselves, but in the runtime environment and the systems that manage them. These include sophisticated CUDA graph management, an asynchronous abstraction for tracking results, and a request path built using the Rust programming language.

Perplexity frames the embedding serving process as two distinct workloads: batch embedding and online embedding. Batch embedding is crucial during the process of building or re-indexing the vector database, where maximizing throughput is key to minimizing costs. Online embedding, conversely, occurs at query time, requiring extremely fast embedding of short queries. A third process, scoring, falls in between these two. After a vector search is performed, large batches of documents are ranked, necessitating a balance between speed and computational resources. A significant architectural decision made by Perplexity was to avoid building a separate, dedicated embedding engine. Recognizing that embedding models are typically small Transformers, the team found that batch embedding operations share similarities with compute-bound prefill stages of large language models (LLMs), while online embedding, often involving very few tokens, resembles memory-bound decode operations. Consequently, Perplexity's research team opted to reuse the prefill and decode kernels already developed for their LLM stack, thereby optimizing resource utilization and development effort.

The request handling process involves three key services: Ivy, Tulip, and ROSE. Ivy functions as a Rust-based HTTP gateway, managing the CPU-side tasks. These include parsing JSON requests, tokenization of input text, input templating, and splitting large batches of requests. Ivy translates incoming requests into a custom gRPC protocol. Furthermore, it is responsible for splitting large-batch requests into smaller chunks and distributing these chunks across available replicas. This load-balancing mechanism is critical for correcting imbalances that can arise when production payloads vary significantly in size, ensuring consistent performance. Tulip serves as the inference server interface. Built using Rust, the tokio asynchronous runtime, and the tonic gRPC framework, Tulip handles the scheduling and batching of requests before dispatching them to the underlying engine. This service acts as a crucial intermediary, orchestrating the flow of data and tasks within the serving infrastructure. ROSE, which stands for Runtime-Optimized Serving Engine, is the component that implements the actual model inference, optimizing for performance and efficiency on the GPU hardware.