Google Cloud has released llm-d 1.0, the first stable version of its distributed inference stack. This open-source, Kubernetes-native layer integrates vLLM or SGLang model servers into one coordinated pool, targeting self-hosted agent and RL workloads where traditional load balancing methods are insufficient.
Key points
- Google Cloud released llm-d 1.0 for distributed LLM inference.
- The stack integrates vLLM or SGLang model servers.
- Performance gains include over 85% TTFT reduction.
- It supports Kubernetes with GPUs or Google TPUs.
- llm-d 1.0 enhances multi-tenant quality of service.
The llm-d 1.0 stack is designed to run on Kubernetes with NVIDIA, AMD, or Intel GPUs, or Google TPUs, utilizing vLLM or SGLang. Performance gains are attributed to cache-aware routing, tiered KV cache offload, and prefill/decode disaggregation. Benchmarks on a 17,000-node multi-region GKE deployment showed a median TTFT reduction of over 85% and a 2.4x throughput increase on an Ironwood TPU coding workload.
llm-d was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, with Red Hat, IBM, and Google Cloud serving as lead maintainers. The project, a CNCF Sandbox project accepted on March 12, 2026, has over 500 contributors from more than 25 organizations and is licensed under Apache 2.0.
By the numbers
- Median TTFT reduction
- 85%
- Throughput increase
- 2.4x
- llm-d version
- 1.0
New features in llm-d 1.0 include multi-tenant quality of service with priority and fairness-aware flow control, a stable foundation with hardened APIs and request paths validated across multiple clouds and accelerators, and zero-downtime upgrades through router high availability and hot standby. It also offers agent-aware serving with session-aware routing and turn-priority scheduling, and async inference capabilities via the llm-d-async-processor.
Sources and references
Original report
Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets
MarkTechPost
Read on MarkTechPost (opens in a new tab)Linked in the original report
- Linkcloud.google.com/blog/topics/ai-infrastructure/llm-d-1-0… (opens in a new tab)
- Linkredhat.com/en/about/press-releases/llm-d-1-advance… (opens in a new tab)
- Linkresearch.ibm.com/blog/llmd-1-0-is-ready-for-production-i… (opens in a new tab)
- Linkpresentations.cncf.io/projects/llm-d (opens in a new tab)
- Codegithub.com/llm-d/llm-d (opens in a new tab)
- Linkllm-d.ai/docs/well-lit-paths/foundations/flow-co… (opens in a new tab)
Ref 00J86M8F
Cite this
APA
Interestana AI Editorial. (2026, October 21). Google Cloud Releases llm-d 1.0 for Distributed LLM Inference. Interestana. https://interestana.com/articles/google-cloud-ships-llm-d-10-with-cache-aware-routing-for-tpu-and-gpu-fleets-00j86m8f
MLA
“Google Cloud Releases llm-d 1.0 for Distributed LLM Inference.” Interestana, 21 Oct. 2026, interestana.com/articles/google-cloud-ships-llm-d-10-with-cache-aware-routing-for-tpu-and-gpu-fleets-00j86m8f.
Chicago
Interestana AI Editorial. “Google Cloud Releases llm-d 1.0 for Distributed LLM Inference.” Interestana, October 21, 2026. https://interestana.com/articles/google-cloud-ships-llm-d-10-with-cache-aware-routing-for-tpu-and-gpu-fleets-00j86m8f.
BibTeX
@misc{interestana_00j86m8f,
author = {{Interestana AI Editorial}},
title = {{Google Cloud Releases llm-d 1.0 for Distributed LLM Inference}},
howpublished = {Interestana},
year = {2026},
month = oct,
url = {https://interestana.com/articles/google-cloud-ships-llm-d-10-with-cache-aware-routing-for-tpu-and-gpu-fleets-00j86m8f},
note = {Summary of reporting by MarkTechPost, https://www.marktechpost.com/2026/10/20/llm-d-1-0-open-source-distributed-inference-stack/}
}Credit the source
Original reporting: MarkTechPost, “Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets”, https://www.marktechpost.com/2026/10/20/llm-d-1-0-open-source-distributed-inference-stack/(opens in a new tab)
Follow this story
Get Computing & Chips in your inbox — new stories in 40 words or fewer, once a week.
Subscribe