KV-cache quantization benchmark
Comparing FP16, INT8, and INT4 KV-cache compression on Llama 3.2-1B with HQQ — memory, throughput, and quality trade-offs.
Comparing FP16, INT8, and INT4 KV-cache compression on Llama 3.2-1B with HQQ — memory, throughput, and quality trade-offs.