Tsinghua University and CUHK researchers have open-sourced AKV, a serving system for KV cache engineering in large language model (LLM) agents. This system allows agents to selectively drop KV cache entries and reuse the remaining ones, rather than summarizing past interactions.
Key points
- Tsinghua AI open-sourced AKV, a KV cache engineering system for LLM agents.
- AKV reduces average cost by 27.7% and increases average task score by 1.7 points.
- The system was tested on GPT-OSS-120B, MiniMax-M2.7, and MiroThinker-1.7-Mini models.
- AKV allows agents to drop and reuse KV cache entries without recomputation.
AKV builds on SGLang and is licensed under Apache-2.0. Testing across three models and four agent benchmarks showed an average cost reduction of 27.7% and an average task score increase of 1.7 points. The system runs on NVIDIA GPUs.
The system was tested on models including GPT-OSS-120B, MiniMax-M2.7, and MiroThinker-1.7-Mini. For GPT-OSS-120B on SWE-bench Verified, AKV reduced cost per task from 20.0 to 9.7 cents and improved the score from 40.0 to 43.2.
By the numbers
- Average cost reduction
- 27.7%
- Average task score increase
- +1.7
- KV cache reads as % of API cost
- 82.4%
Cache engineering, as implemented by AKV, differs from context engineering by leaving the text trajectory untouched. Instead, it modifies which KV entries are visible to the attention mechanism. Retained entries are reused without recomputation, potentially retaining information from dropped messages.
KV cache reads constituted 82.4% of total API costs in the research paper's analysis. AKV defines three cache operations: Append for standard updates, Drop to remove selected entries without altering positions, and Repos to rewrite retained keys to specific positions using RoPE rotation.
Sources and references
Original report
Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang
MarkTechPost
Read on MarkTechPost (opens in a new tab)Linked in the original report
- Codegithub.com/thu-nics/AKV (opens in a new tab)
- Paperarxiv.org/abs/2610.28577 (opens in a new tab)
- Codegithub.com/thu-nics/AKV/blob/main/LICENSE (opens in a new tab)
- Codegithub.com/sgl-project/sglang (opens in a new tab)
- Paperarxiv.org/html/2610.28577v1 (opens in a new tab)
- Paperarxiv.org/abs/2104.09864 (opens in a new tab)
Ref 004XTM1Q
Cite this
APA
Interestana AI Editorial. (2026, October 26). Tsinghua AI Open-Sources AKV Cache System for LLM Agents. Interestana. https://interestana.com/articles/tsinghua-ai-open-sources-akv-a-cache-engineering-server-for-llm-agents-on-sglang-004xtm1q
MLA
“Tsinghua AI Open-Sources AKV Cache System for LLM Agents.” Interestana, 26 Oct. 2026, interestana.com/articles/tsinghua-ai-open-sources-akv-a-cache-engineering-server-for-llm-agents-on-sglang-004xtm1q.
Chicago
Interestana AI Editorial. “Tsinghua AI Open-Sources AKV Cache System for LLM Agents.” Interestana, October 26, 2026. https://interestana.com/articles/tsinghua-ai-open-sources-akv-a-cache-engineering-server-for-llm-agents-on-sglang-004xtm1q.
BibTeX
@misc{interestana_004xtm1q,
author = {{Interestana AI Editorial}},
title = {{Tsinghua AI Open-Sources AKV Cache System for LLM Agents}},
howpublished = {Interestana},
year = {2026},
month = oct,
url = {https://interestana.com/articles/tsinghua-ai-open-sources-akv-a-cache-engineering-server-for-llm-agents-on-sglang-004xtm1q},
note = {Summary of reporting by MarkTechPost, https://www.marktechpost.com/2026/10/26/tsinghua-akv-kv-cache-engineering-llm-agents/}
}Credit the source
Original reporting: MarkTechPost, “Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang”, https://www.marktechpost.com/2026/10/26/tsinghua-akv-kv-cache-engineering-llm-agents/(opens in a new tab)
Follow this story
Get Artificial Intelligence in your inbox — new stories in 40 words or fewer, once a week.
Subscribe