The first thing to understand is that KV cache compression is not a brand-new concept that suddenly appeared out of nowhere; this direction has been in motion for a while, and every serious AI lab has already been aggressively compressing memory during inference. What Google did with TurboQuant is meaningful from an engineering standpoint, but […]