TurboQuant is Google Research's training-free KV cache compression method for LLM inference that combines PolarQuant rotation-plus-scalar-quantization with a 1-bit QJL residual correction to deliver near-lossless 3-bit KV cache compression and up to 8× faster attention on H100 GPUs.
What is TurboQuant?
TurboQuant is a two-stage KV cache quantization method developed by Google Research and accepted at ICLR 2026 (with its PolarQuant component accepted at AISTATS 2026). It takes the FP16 key/value vectors cached during transformer inference and compresses them to 3–4 bits per channel using polar-coordinate rotation, scalar quantization, and a 1-bit Quantized Johnson-Lindenstrauss residual correction. The output is a compressed KV cache that can be dequantized for attention computation, running on any CUDA-capable GPU and requiring no training or calibration data. The paper is available on arXiv (2504.19874).
Key Features
- Two-stage compression — Stage one, PolarQuant, rotates each vector with a random orthogonal matrix, scalar-quantizes the rotated coordinates against Lloyd-Max centroids, and stores the original norm. Stage two applies a 1-bit QJL sketch to the residual, restoring unbiased inner-product estimates for attention.
- No training or fine-tuning — The method is data-oblivious and applied at inference time to any transformer model.
- 3–4 bit per-channel compression — Google reports at least 6× KV cache memory reduction from FP16, with quality matching FP16 at roughly 3.5 bits per channel and measurable loss only below about 2.5 bits.
- Up to 8× attention speedup — Measured on NVIDIA H100 in 4-bit mode; speedup depends on hardware and kernel implementation.
- Strong benchmark results — TurboQuant outperforms KIVI on LongBench aggregate scores across Gemma and Mistral families and maintains full needle-in-a-haystack retrieval with a score of 0.997, vs 0.981 for KIVI and 0.858 for SnapKV.
- Works with existing stacks — It is compatible with weight quantization methods like GPTQ/AWQ and can be combined with paged attention. Community PyTorch and vLLM integrations exist, and an interactive KV cache calculator on the TurboQuant site supports MHA, GQA, and MQA shapes.
Who is it for?
- Long-context serving teams: RAG pipelines, document Q&A, and multi-turn chat at 32k–128k tokens per request, where the KV cache becomes the dominant memory cost.
- High-throughput inference operators: Batch serving with many concurrent requests; cutting per-request cache from FP16 to 3 bits lets more requests fit on the same GPU.
- Consumer GPU users: Running 70B-class models on 24 GB cards; weight quantization alone may not free enough memory, and 3-bit cache compression can close the gap.
- Vector search developers: The same PolarQuant + QJL pipeline applies to vector search workloads as well as transformer inference.
What can you do with TurboQuant?
- Serve a Llama 3.1 70B request at 128k context: Reduce estimated KV cache from about 40 GB in FP16 to about 7.5 GB at 3 bits.
- Expand batch size: Free up GPU memory that would otherwise be consumed by FP16 key/value caches, enabling more concurrent users per GPU.
- Deploy long-context models locally: Run models like Qwen 2.5 32B or DeepSeek-R1 with 128k context on memory budgets that would otherwise require multiple GPUs.
How does TurboQuant work?
- PolarQuant stage: Multiply each key/value vector by a random orthogonal matrix, quantize each rotated coordinate to the nearest precomputed Lloyd-Max centroid, and store the vector norm.
- QJL residual stage: Compute the residual between the rotated vector and its dequantized form, multiply by a random sign matrix, and keep a 1-bit sign sketch.
- At decode time: Dequantize the scalar codes and add the QJL decode of the sign sketch to reconstruct an approximate vector for attention.
Pros and cons
- Pros: No training data needed; near-lossless quality at 3.5 bits per channel; 6× memory reduction; up to 8× attention speedup on H100; complementary to weight quantization.
- Cons: Not yet officially merged into llama.cpp or vLLM as of April 2026; there is no official Google reference implementation; quality degrades measurably under about 2.5 bits per channel.
Alternatives
- KIVI: A 2-bit per-channel/per-token KV cache quantizer with asymmetric quantization and no residual correction; more aggressive compression but a lower needle score of 0.981.
- SnapKV: Evicts low-attention tokens instead of quantizing them; faster generation but lossy, with a needle score of 0.858.
- GPTQ / AWQ: Weight quantization methods that are complementary rather than competing; they compress model weights while TurboQuant compresses the cache.
- PagedAttention: A memory-layout manager for KV cache paging; orthogonal to quantization and can be used together with TurboQuant.
FAQ
Does TurboQuant require training or fine-tuning?
No. TurboQuant is data-oblivious and applied at inference time. It needs no training data, calibration set, or model-specific tuning, and it works on any transformer model.
How much memory does TurboQuant save?
Google reports at least 6× KV cache memory reduction compared to FP16. For example, a Llama 3.1 70B request at 128k context uses about 40 GB of FP16 KV cache; at 3 bits, TurboQuant would reduce that to roughly 7.5 GB.
Is the 'zero-loss' claim accurate?
Quality parity with FP16 is reached at approximately 3.5 bits per channel on LongBench and needle-in-a-haystack benchmarks. Below about 2.5 bits per channel, measurable accuracy loss appears. The '3-bit zero-loss' headline simplifies this continuous precision tradeoff.
Is TurboQuant available in vLLM or llama.cpp?
Not officially as of 2026-04-19. Community PyTorch and vLLM integrations exist, and llama.cpp tracks support in Issue #20977 and PR #21089, which propose tbq3_0 / tbq4_0 KV cache types. Ollama and LM Studio status are tracked on the same page.
Can I combine TurboQuant with weight quantization?
Yes. TurboQuant compresses the KV cache, while GPTQ or AWQ compress model weights. The two approaches are complementary and can be stacked for maximum memory savings.
