Gigatoken Rust Tokenizer Achieves 24.53 GB/s Speed

Gigatoken, a new byte-pair encoding (BPE) tokenizer developed in Rust with Python bindings, has demonstrated unprecedented text processing speeds, achieving 24.53 GB/s on a 144-core AMD EPYC 9565 dual-socket setup. This performance represents a 989x speedup compared to HuggingFace tokenizers and a 681x improvement over OpenAI's tiktoken when processing the 11.9 GB owt_train.txt corpus using the GPT-2 tokenizer.

The library, released under an MIT license by PhD student Marcel Rød from Stanford, aims to address the often-overlooked performance bottlenecks in the language modeling stack. On an Apple M4 Max with 16 cores, Gigatoken achieved 8.79 GB/s, outperforming HuggingFace tokenizers by 1,268x and tiktoken by 140x. Even on a consumer AMD Ryzen 7 9800X3D, it reached 6.27 GB/s, demonstrating substantial gains over established tools.

Gigatoken supports a wide array of tokenizer families, including GPT-2, GPT-OSS, Llama 3 through 4, Qwen 2 through 3.6, DeepSeek V3/R1/V4, GLM 4 and 5, Kimi K2, Nemotron 3, Phi-4, OLMo 2/3, ModernBERT, Gemma, and Mistral. It offers two usage modes: a compatibility mode that wraps existing HuggingFace or tiktoken tokenizers to ensure output parity, albeit with a throughput cost, and a native mode that leverages its optimized Rust implementation.

The library is available on PyPI as 'gigatoken' (version 0.9.0, released July 21, 2026) and can be installed via pip. The project's codebase is composed of 66.2% Rust and 33.3% Python, highlighting its foundation in high-performance systems programming.