Google présente TurboQuant : un nouvel algorithme de compression qui réduit de 6x la mémoire cache clé-valeur des LLM et offre jusqu'à 8x d'accélération, sans aucune perte de précision

The scaling of Large Language Models (LLMs) is increasingly constrained by memory communication overhead between High-Bandwidth Memory (HBM) and SRAM. Specifically, the Key-Value (KV) cache size scales with both model dimensions and context length, creating a significant bottleneck for long-context inference. Google research team has proposed TurboQuant , a data-oblivious quantization framework designed to achieve near-optimal distortion rates for high-dimensional Euclidean vectors while addressing both mean-squared error (MSE) and inner product distortion. Addressing the Memory Wall with Data-Oblivious VQ Vector quantization (VQ) in Euclidean space is a foundational problem rooted in Shannon’s source coding theory . Traditional VQ algorithms, such as Product Quantization (PQ), often require extensive offline preprocessing and data-dependent codebook training, making them ill-suited for the dynamic requirements of real-time AI workloads like KV cache management . TurboQuant is a ‘data-oblivious’ algorithm and it does not require dataset-specific tuning or calibrations. It is designed to be highly compatible with modern accelerators like GPUs by leveraging vectorized operations rather than slow, non-parallelizable binary searches. The Geometric Mechanics of TurboQuant The core mechanism of TurboQuant involves applying a random rotation Π E R d x d to the input vectors. This rotation induces a concentrated Beta distribution on each coordinate, regardless of the original input data. In high dimensions, these coordinates become nearly independent and identically distributed (i.i.d.). This near-independence simplifies the quantization design, allowing TurboQuant to solve a continuous 1D k-means / Max-Lloyd scalar quantization problem per coordinate. The optimal scalar quantizer for a given bit-width b is found by minimizing the following MSE cost function: $$\mathcal{C}(f_{X},b):=min_{-1\le c_{1}\le c_{2}\le…\le c_{2^{b}}\le1}\sum_{i=1}^{2^{b}}\int_{\frac{c_{i-1}+c_{i}}{2}}^{\frac{c_{i}+c_{i+1}}{2}}|x-c_{i}|^{2}\cdot f_{X}(x)dx$$ <> By solving this optimization once for relevant bit-widths and storing the resulting codebooks, TurboQuant can efficiently quantize vectors during online inference . Eliminating Inner Product Bias A primary challenge in quantization is that maps optimized strictly for MSE often introduce bias when estimating inner products, which are the fundamental operations in transformer attention mechanisms. For example, a 1-bit MSE-optimal quantizer in high dimensions can exhibit a multiplicative bias of 2/π. To correct this, Google Research developed TURBOQUANT prod , a two-stage approach : MSE Stage : It applies a TURBOQUANT mse quantizer using a bit-width of b-1 to minimize the L 2 norm of the residual vector. Unbiased Stage : It applies a 1-bit Quantized Johnson-Lindenstrauss (QJL) transform to the residual vector. This combination results in an overall bit-width of b while providing a provably unbiased estimator for inner products: \(\mathbb{E}_{Q}[\langle y,Q^{-1}(Q(x))\rangle ]=\langle y,x\rangle \) Theoretical and Empirical Performance The research team established information-theoretic lower bounds using Shannon’s Lower Bound (SLB) and Yao’s minimax principle. TurboQuant’s MSE distortion is provably within a small constant factor (≈ 2.7) of the absolute theoretical limit across all bit-widths. At a bit-width of b =1, it is only a factor of approximately 1.45 away from the optimal. Bit-width (b) TURBOQUANT mse​ Distortion Information-Theoretic Lower Bound 1 0.36 0.25 2 0.117 0.0625 3 0.03 0.0156 4 0.009 0.0039 In end-to-end LLM generation benchmarks using Llama-3.1-8B-Instruct and Ministral-7B-Instruct , TurboQuant demonstrated high quality retention . Under a 4x compression ratio, the model maintained 100% retrieval accuracy on the Needle-In-A-Haystack benchmark . In the Needle-In-A-Haystack benchmark, TurboQuant matched full-precision performance up to 104k tokens under 4× compression . For non-integer bit-widths, the system employs an outlier treatment strategy, allocating higher precision (e.g., 3 bits) to specific outlier channels and lower precision (e.g., 2 bits) to non-outliers, resulting in effective bit-rates like 2.5 or 3.5 bits per channel . https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/ Speed and Indexing Efficiency In nearest neighbor search tasks, TurboQuant outperformed standard Product Quantization (PQ) and RabitQ in recall while reducing indexing time to virtually zero . Because TurboQuant is data-oblivious, it eliminates the need for the time-consuming k-means training phase required by PQ, which can take hundreds of seconds for large datasets . Approach d=200 Indexing d=1536 Indexing d=3072 Indexing Product Quantization 37.04s 239.75s 494.42s TurboQuant 0.0007s 0.0013s 0.0021s TurboQuant represents a mathematically grounded shift toward efficient, hardware-compatible vector quantization that bridges the gap between