LLM Model Formats: GGUF, GPTQ, AWQ, EXL2 Explained

The landscape of large language model (LLM) formats can be confusing due to the conflation of two distinct concepts: tensor storage containers and quantization methods. A tensor storage container defines how model weights are organized and saved on disk, while a quantization method dictates how these weights are compressed into fewer bits to reduce memory footprint and potentially increase inference speed. Common container formats include safetensors, GGUF, and PyTorch's pickle format (.bin/.pt). Quantization methods encompass GPTQ, AWQ, bitsandbytes NF4, and llama.cpp's K-quants and I-quants. Some formats, like EXL2 and EXL3, combine a specific quantization method with a storage layout optimized for a particular inference library.

A fundamental rule of thumb for estimating weight memory is: Weight memory ≈ parameters × bits-per-weight ÷ 8. For instance, an 8 billion parameter model with 16-bit weights would require approximately 16 GB of memory for its weights, while a 70 billion parameter model would need around 140 GB. This calculation pertains solely to the model weights; additional memory is consumed by the KV cache and runtime overhead during inference. Full precision models, typically stored as 16-bit weights, are often distributed in PyTorch's .bin or .pt files, which utilize Python's pickle mechanism. However, loading untrusted pickle files poses a security risk due to their ability to execute arbitrary code. Safetensors, developed at Hugging Face and now a PyTorch Foundation project, mitigates this risk by storing tensors as raw data buffers preceded by a small JSON header, making them non-executable and enabling memory-mapping for efficient loading of individual tensors without reading the entire file. It is important to note that many models quantized using methods like GPTQ, AWQ, EXL2, EXL3, and MLX are still packaged within .safetensors files, with the quantization details residing in the tensor data and a configuration file rather than a new container format.

GGUF represents a specific binary format designed for running models using GGML and its derivatives, such as the popular llama.cpp inference engine. This format was created by Georgi Gerganov, the lead developer of llama.cpp. GGUF was introduced on August 21, 2023, succeeding the older GGML format. The transition from GGML to GGUF was driven by limitations in the earlier format, which lacked the ability to specify a model's architecture. This deficiency meant that introducing a new hyperparameter could break compatibility with all existing GGML files. GGUF addresses this by adopting a typed key-value metadata system, allowing for more robust and extensible model descriptions. This evolution in model format is crucial for efficient deployment and execution of LLMs across various hardware configurations, enabling broader accessibility and performance optimization.