Meta Unveils ZGateway Proxy for ZippyDB, Handling 1 Billion Ops/Sec

Meta's engineering team has introduced ZGateway, a new stateless proxy tier designed to sit between client applications and ZippyDB, Meta's primary key-value store. ZippyDB is a critical infrastructure component that supports product metadata, counters, and configuration, processing billions of operations per second across Meta's services. ZGateway was initially developed to address the issue of connection sprawl, which affected over a million client hosts, and has since evolved to incorporate essential functionalities such as batching, admission control, caching, and failover mechanisms.

Prior to ZGateway, direct access to ZippyDB meant that each client application had to establish connections to every database host it required. This architecture led to a significant number of TLS connections, with a single client potentially interacting with tens of thousands of shards across hundreds of thousands of hosts. Both typical clients and database hosts were burdened with tens of thousands of TLS connections. Each idle connection consumed valuable memory, CPU resources, and file descriptors on both ends, and the number of inbound connections escalated with every new client cohort. This complexity resulted in severe operational challenges, including reconnection storms that caused system crashes due to file descriptor exhaustion and Out-Of-Memory (OOM) errors. One notable incident involved a routing bug that prompted every client to open a connection per shard, leading the entire fleet into a reboot loop. Implementing client-side fixes proved impractical due to the distributed ownership of the client fleet among hundreds of teams.

ZGateway functions as a stateless proxy tier situated between ZippyDB clients and the ZServer database fleet. According to Meta, this new tier manages over 1 billion operations per second and currently handles approximately 40% of ZippyDB traffic, with projections indicating it will soon exceed 60%. This is achieved with a computational overhead of about 6% for an average use case. The infrastructure for ZGateway is deployed as regional tiers, discovered via ServiceRouter, Meta's service mesh. It operates in two primary configurations: a pure proxy and a read-through cache. The underlying engine of ZGateway is Meta's robust C++ ZippyDB client, effectively making ZGateway a managed service version of the ZippyDB client. In this setup, a client initiates requests through a sticky connection to a regional ZGateway host. This host then terminates TLS, performs authorization against the use case's Access Control Lists (ACLs), enforces per-tenant admission control and traffic shaping, resolves the appropriate shard, checks the local cache on caching tiers, and batches the request with other ongoing operations destined for that specific shard.