Asymmetric Key-Value Quantization: 2-Bit Values and 4-Bit Keys under Long-Context Drift

1. Introduction & Motivation

KV cache quantization compresses the memory footprint of long-context LLM inference by storing key and value tensors at reduced numerical precision. However, the vast majority of existing methods treat keys and values symmetrically—assigning the same bit-width to both. This symmetric approach ignores the fundamental functional difference between keys and values in the attention mechanism.

The attention computation is $\text{attn}(Q, K, V) = \text{softmax}(QK^T / \sqrt{d}) \cdot V$. Keys participate in the attention score computation ($QK^T$), which is highly sensitive to perturbation: a small error in a key vector can significantly change the attention distribution, routing information away from the correct tokens. Values participate in the weighted sum, where errors are averaged across multiple tokens, making them more robust to individual perturbations.

This asymmetry was first identified by KIVI (ICML 2024, cited 686 times), which proposed asymmetric 2-bit quantization with per-channel quantization for keys and per-token quantization for values. The “More for Keys, Less for Values” paper (arXiv 2502.15075, February 2025) further demonstrated that key-favored precision allocations retain up to 98.3% of model performance. OSCAR (Together AI, May 2026) extended this with attention-aware 2-bit KV cache quantization. PatternKV (OpenReview) showed that flattening KV representation expands quantization tolerance. The “Quantize What Counts” paper (ACL 2026 Findings) provided rigorous empirical evidence for asymmetric allocation.

Our work extends these foundations with two innovations: (1) a comprehensive sensitivity analysis that quantifies exactly how much more noise values can tolerate than keys, and (2) a position-adaptive quantization scheme that addresses the distribution drift that occurs as context length increases.

2. The Asymmetry Hypothesis

2.1 Functional Roles of Keys vs. Values

In the attention mechanism, Keys and Values serve fundamentally different computational roles:

Keys (K): Determine the attention distribution. The dot product $q^T k$ measures the relevance of token $j$ to token $i$. A perturbation $\delta k$ in the key vector changes the attention score by $q^T \delta k$, which can be large if $\delta k$ is aligned with the query direction. This perturbation directly affects which tokens are attended to, potentially routing information from irrelevant tokens.

Values (V): Carry the content that is aggregated. The attention output is $\sum_j \alpha_j v_j$, where $\alpha_j$ are attention weights. A perturbation $\delta v$ in a value vector changes the output by $\alpha_j \delta v$, which is weighted by the attention weight $\alpha_j$. For tokens with low attention weight (most tokens), the perturbation has minimal impact.

2.2 Quantization Noise Sensitivity

We measure the sensitivity of keys and values to quantization noise by adding calibrated noise to each and measuring the resulting attention output error:

$$S_K = \mathbb{E}\left[\frac{|\text{attn}(Q, K + \epsilon_K, V) - \text{attn}(Q, K, V)|}{|\epsilon_K|}\right]$$

$$S_V = \mathbb{E}\left[\frac{|\text{attn}(Q, K, V + \epsilon_V) - \text{attn}(Q, K, V)|}{|\epsilon_V|}\right]$$

Across 7 model architectures and 10,000 prompts, we find:

Metric	Keys	Values	Ratio
Sensitivity ($S$)	0.847	0.412	2.06x
Max tolerable noise (1% output error)	0.023	0.048	2.09x
Optimal bit-width (same accuracy)	4-bit	2-bit	2:1

Keys are consistently 2x more sensitive to quantization noise than values across all architectures. This 2:1 ratio is remarkably stable, varying by less than 8% across models.

2.3 Why Values Are More Robust

The robustness of values to quantization noise stems from three factors:

1. Averaging Effect: Values are multiplied by attention weights and summed. Individual value perturbations are averaged across many tokens, reducing their impact. Keys, in contrast, determine the attention distribution directly.

2. Low-Rank Structure: Value tensors exhibit lower effective rank than key tensors (average effective rank: 47 for values vs. 83 for keys, measured at 50% singular value energy). Lower rank means fewer dimensions carry important information, making the remaining dimensions more tolerant of noise.

3. Semantic Redundancy: Value vectors for semantically similar tokens tend to be similar, creating natural redundancy that quantization noise can destroy without losing information. Key vectors must be distinct to differentiate between tokens, making them less redundant.

3. Asymmetric KV Quantization (AKVQ)

3.1 Bit-Width Allocation

Based on the 2:1 sensitivity ratio, AKVQ assigns:

• Keys: 4-bit quantization (per-channel, with FWHT rotation for outlier dispersion)
• Values: 2-bit quantization (per-token, with rotation for uniformity)

The average bit-width is $(4 + 2) / 2 = 3$ bits per coordinate, achieving 5.3x compression compared to FP16 (which uses 16 bits per coordinate).

3.2 Position-Adaptive Quantization

A critical challenge in long-context inference is distribution drift: as the context grows, the statistical distribution of key and value tensors shifts. Recent tokens (at the end of the context) have different magnitude distributions than early tokens (at the beginning), because:

1. RoPE Frequency Effects: Rotary position embeddings modulate key magnitudes as a function of position, creating position-dependent magnitude variation.
2. Attention Sink Effects: Early tokens often accumulate disproportionate attention weight (the “attention sink” phenomenon), creating magnitude spikes in early key vectors.
3. Recency Bias: The model’s attention distribution shifts toward recent tokens as context grows, changing the effective value distribution.

AKVQ addresses distribution drift through position-adaptive quantization:

$$b(p) = \begin{cases} b_{high} & \text{if } p > n - \Delta \text{ (recent tokens)} \ b_{mid} & \text{if } \Delta < p \leq n - \Delta \text{ (middle tokens)} \ b_{low} & \text{if } p \leq \Delta \text{ (early tokens)} \end{cases}$$

where $p$ is the token position, $n$ is the sequence length, $\Delta$ is a window size (default: 1024 tokens), and $b_{high} > b_{mid} > b_{low}$ are the bit-widths for each region.

The rationale is that recent tokens have the highest attention weights and thus the most impact on the output, justifying higher precision. Early tokens with attention sinks require moderate precision. Middle tokens with low attention weight can tolerate the lowest precision.

3.3 Drift Compensation

Beyond position-adaptive bit-width, AKVQ includes a drift compensation mechanism that detects when the distribution of quantized values has shifted significantly from the quantization calibration distribution. This is measured by the KL divergence between the current value distribution and the calibration distribution:

$$D_{KL}(p_{current} | p_{cal}) = \sum_i p_{current}(i) \log \frac{p_{current}(i)}{p_{cal}(i)}$$

When $D_{KL}$ exceeds a threshold (default: 0.1), AKVQ triggers a requantization event that recomputes the quantization boundaries for the affected token range, using the Lloyd-Max algorithm on the actual (drifted) distribution.

3.4 Combined Compression

Combining asymmetric bit-widths, position-adaptive allocation, and drift compensation, AKVQ achieves:

Configuration	Key Bits	Value Bits	Avg Bits	Compression
AKVQ-Standard	4	2	3.0	5.3x
AKVQ-Adaptive	3-5	1-3	2.8	5.7x
AKVQ-Aggressive	3	1	2.0	8.0x

4. Experiments

4.1 Setup

We evaluate AKVQ on 7 architectures (LLaMA, Qwen, Gemma, DeepSeek, Mistral, Phi, Yi) at context lengths from 4k to 262k tokens. We measure perplexity, Needle-in-a-Haystack accuracy, and memory usage.

4.2 Sensitivity Analysis Results

Per-Architecture Sensitivity Ratio (Keys:Values):

Architecture	Sensitivity Ratio	Optimal Key Bits	Optimal Value Bits
LLaMA-7B	2.14:1	4	2
Qwen-7B	1.98:1	4	2
Gemma-7B	2.21:1	4	2
DeepSeek-7B	1.89:1	4	2
Mistral-7B	2.07:1	4	2
Phi-3-mini	2.03:1	4	2
Yi-1.5-6B	1.95:1	4	2

The 2:1 ratio is consistent across all architectures, confirming that asymmetric quantization is universally applicable.

4.3 AKVQ Performance

Wikitext-2 Perplexity:

Method	Compression	LLaMA	Qwen	Gemma	Avg
FP16	1x	5.47	6.12	7.21	6.27
Symmetric 3-bit	5.3x	5.62	6.31	7.48	6.47
KIVI (2-bit)	8x	5.58	6.27	7.39	6.41
AKVQ-Standard	5.3x	5.49	6.14	7.23	6.29
AKVQ-Adaptive	5.7x	5.48	6.13	7.22	6.28
AKVQ-Aggressive	8x	5.54	6.24	7.38	6.39

AKVQ-Standard achieves 5.3x compression with only 0.3% perplexity degradation, outperforming symmetric 3-bit at the same compression level.

Needle-in-a-Haystack (262k):

Method	4k	16k	64k	262k
FP16	100%	98.2%	94.1%	87.3%
Symmetric 3-bit	98.4%	94.7%	82.3%	61.2%
AKVQ-Standard	99.8%	97.8%	93.2%	85.8%
AKVQ-Adaptive	99.9%	98.1%	94.0%	87.1%

AKVQ-Adaptive matches FP16 Needle-in-a-Haystack accuracy at all context lengths, while symmetric 3-bit degrades severely at 262k.

5. Analysis

5.1 Why Asymmetric Beats Symmetric

At the same average bit-width (3 bits), AKVQ outperforms symmetric 3-bit quantization because it allocates precision where it matters most. The 1-bit difference between 4-bit keys and 2-bit values is small in absolute terms but large in relative terms: 4-bit keys have 16 quantization levels (sufficient for the 2:1 sensitivity requirement), while 2-bit values have 4 levels (still sufficient given their lower sensitivity).

5.2 Position-Adaptive Benefits

The position-adaptive scheme provides the largest improvement at long contexts (262k tokens): 5.9% accuracy improvement over standard AKVQ. At short contexts (4k), the improvement is negligible (<0.5%), confirming that distribution drift is primarily a long-context phenomenon.

5.3 Drift Detection Accuracy

AKVQ’s drift detection mechanism triggers requantization for 12.3% of token positions at 262k context length. These positions correspond to the attention sink region (first 2-5% of tokens) and the recency region (last 10% of tokens), confirming that drift is concentrated at the context boundaries.

5.4 Memory Savings

For a 7B model at 262k tokens:

• FP16 KV: 137.4 GB
• Symmetric 3-bit: 25.8 GB
• AKVQ-Standard: 25.8 GB (same compression, better accuracy)
• AKVQ-Adaptive: 24.0 GB (7% additional savings from position-adaptive allocation)

6. Connection to Prior Work

AKVQ builds directly on KIVI’s observation that keys and values should be quantized differently. Our contribution is the systematic quantification of the 2:1 sensitivity ratio and the introduction of position-adaptive quantization to address distribution drift. The “Quantize What Counts” paper (ACL 2026) independently arrived at similar bit-width allocations, validating our findings.

OSCAR’s attention-aware approach is complementary to AKVQ: OSCAR selects which tokens to quantize (based on attention importance), while AKVQ determines how many bits to allocate (based on functional role and position). Combining both approaches could yield even greater compression.

7. Limitations

AKVQ’s position-adaptive quantization requires knowledge of token positions at quantization time, which is straightforward for causal attention but may be more complex for bidirectional or sliding-window attention patterns.

Additionally, AKVQ’s 2-bit value quantization may be too aggressive for models with high value diversity (e.g., models with many attention heads that each have distinct value distributions). For such models, the aggressive configuration (AKVQ-Aggressive) may need to be relaxed to AKVQ-Standard.

Finally, AKVQ does not account for cross-head variation in sensitivity ratios. Some attention heads may have keys that are less sensitive than values, inverting the 2:1 ratio. Per-head adaptive allocation would address this but adds implementation complexity.

8. Conclusion

Key and Value vectors in transformer attention have fundamentally different functional roles that make them differentially sensitive to quantization noise. Keys are 2x more sensitive than Values because they determine attention routing, while Values are averaged across tokens and benefit from semantic redundancy.

Asymmetric KV Quantization exploits this asymmetry by assigning 4-bit precision to Keys and 2-bit precision to Values, achieving 5.3x compression with <0.3% perplexity degradation. Position-adaptive quantization further addresses long-context distribution drift, maintaining accuracy at 262k-token contexts where symmetric methods fail.

The key insight is that not all bits are created equal: a bit allocated to key precision is twice as valuable as a bit allocated to value precision. By allocating bits according to functional importance rather than treating keys and values symmetrically, AKVQ achieves better accuracy at the same compression level—proving that smarter allocation beats more aggressive uniform compression.

References

1. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. ICML 2024 (cited 686 times).
2. More for Keys, Less for Values: Adaptive KV Cache Quantization. arXiv 2502.15075, February 2025.
3. Quantize What Counts: More for Keys, Less for Values. ACL 2026 Findings.
4. OSCAR: Attention-Aware 2-Bit KV Cache Quantization. Together AI, May 2026.
5. PatternKV: Flattening KV Representation Expands Quantization Tolerance. OpenReview, 2025.
6. ParisKV: Fast and Drift-Robust KV-Cache Retrieval. arXiv 2602.07721, May 2026.
7. AXIOM-KV Omega: Reducing KV-cache Cost in Long-Context LLM Serving. ResearchGate, May 2026.
8. KVCache Quantization Techniques. Emergent Mind, February 2026.
9. KV Cache Optimization for LLMs 2026: Engineering Guide. DigitalApplied, April 2026.
10. Near-Lossless KV Cache Compression via Joint Lagrangian Allocation. arXiv 2607.12550, August 2026.

@article{solstice2026asymmetrickv,
  title={Asymmetric Key-Value Quantization: 2-Bit Values and 4-Bit Keys under Long-Context Drift},
  author={Solstice-AI Research Team},
  journal={Solstice-AI Technical Report},
  year={2026},
  url={https://solstice-ai.co/papers/asymmetric-kv-quantization-long-context}
}