KV cache might be a bigger problem for local models than parameter count

Everyone keeps talking about fitting larger models into local hardware, but parameter count isn’t the whole story.

For long context inference, KV cache can become the real memory bottleneck. Every new token adds key and value states that need to stay available, so a model that fits comfortably in VRAM can suddenly become painful at 100k or 200k context.

GQA and MQA help by reducing KV heads, and KV cache quantization helps even more, but the cache still grows with context.

This makes me think future local model optimization will be less about simply reducing parameter count and more about reducing memory movement and persistent state.

The interesting question is whether we eventually optimize models around "how much they need to remember", rather than just how many parameters they have.