Researchers from Stanford and UC Berkeley have introduced TokenGate, a module designed to improve long-context prefill in large language models (LLMs). TokenGate employs learned token pruning, allowing LLMs to process only the most relevant tokens for full processing, thereby skimming the input.
Key points
- TokenGate speeds up LLM long-context prefill by pruning tokens.
- Achieves up to 11.3x faster time-to-first-token.
- Reduces prefill memory usage by 1.5x to 3.4x.
- Maintains accuracy while pruning tokens.
- Code is available on GitHub under Apache-2.0.
The research team reports that TokenGate can achieve a time-to-first-token (TTFT) speedup of 1.4x to 11.3x without any loss in accuracy. Additionally, peak prefill memory usage is reduced by 1.5x to 3.4x. The code and checkpoints for TokenGate are available on GitHub under the Apache-2.0 license.
TokenGate functions as a drop-in component that adds trainable token pruning to pretrained LLMs. Unlike standard transformer architectures where every token is processed through every layer, TokenGate uses a gate module. This module, a small MLP, scores each token at a specific layer (L). A top-K selector then keeps the highest-scoring tokens for further processing by layers above L.
By the numbers
- Faster time-to-first-token
- 11.3x
- Lower memory usage
- 3.4x
- Context length tested
- 128k
This approach is particularly effective for the document part of a prompt, while the question itself is never pruned. TokenGate utilizes a technique similar to Mixture-of-Experts routers to enable end-to-end training. Each kept token's score acts as a bias on attention logits, providing gradients for the scorer. A learnable per-head gain and temperature refine this process.
Performance evaluations show TokenGate is Pareto-optimal on five tasks compared to six baselines. The best results include a 11.3x faster TTFT and 3.42x lower memory usage on the NIAH 16k task. At a context length of 128k tokens, the TTFT gain can reach 13.0x, though speedups drop to 1.4x on some tasks.
Sources and references
Original report
Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster
MarkTechPost
Read on MarkTechPost (opens in a new tab)Linked in the original report
- Paperarxiv.org/abs/2610.26754 (opens in a new tab)
- Codegithub.com/token-gate-llms/TokenGate (opens in a new tab)
- Paperarxiv.org/abs/2502.01068 (opens in a new tab)
- Paperarxiv.org/abs/2508.06447 (opens in a new tab)
- Paperarxiv.org/abs/2409.17422 (opens in a new tab)
- Paperarxiv.org/abs/2604.18103 (opens in a new tab)
Ref 00OGHMM6
Cite this
APA
Interestana AI Editorial. (2026, October 25). TokenGate Speeds Up LLM Long-Context Prefill. Interestana. https://interestana.com/articles/researchers-from-stanford-and-uc-berkeley-introduce-tokengate-token-gated-attention-makes-long-context-prefill-up-to-113x-faster-00oghmm6
MLA
“TokenGate Speeds Up LLM Long-Context Prefill.” Interestana, 25 Oct. 2026, interestana.com/articles/researchers-from-stanford-and-uc-berkeley-introduce-tokengate-token-gated-attention-makes-long-context-prefill-up-to-113x-faster-00oghmm6.
Chicago
Interestana AI Editorial. “TokenGate Speeds Up LLM Long-Context Prefill.” Interestana, October 25, 2026. https://interestana.com/articles/researchers-from-stanford-and-uc-berkeley-introduce-tokengate-token-gated-attention-makes-long-context-prefill-up-to-113x-faster-00oghmm6.
BibTeX
@misc{interestana_00oghmm6,
author = {{Interestana AI Editorial}},
title = {{TokenGate Speeds Up LLM Long-Context Prefill}},
howpublished = {Interestana},
year = {2026},
month = oct,
url = {https://interestana.com/articles/researchers-from-stanford-and-uc-berkeley-introduce-tokengate-token-gated-attention-makes-long-context-prefill-up-to-113x-faster-00oghmm6},
note = {Summary of reporting by MarkTechPost, https://www.marktechpost.com/2026/10/25/stanford-berkeley-tokengate-learned-token-pruning-llm-prefill/}
}Credit the source
Original reporting: MarkTechPost, “Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster”, https://www.marktechpost.com/2026/10/25/stanford-berkeley-tokengate-learned-token-pruning-llm-prefill/(opens in a new tab)
Follow this story
Get Artificial Intelligence in your inbox — new stories in 40 words or fewer, once a week.
Subscribe