Is llama.cpp meant to be slow at long context, even when you aren't using that context?

I am trying out a few fine-tunes of Qwen 3.5 9B @ IQ4_XS @ 131K context and trying to go mostly local (free beats cheap, after all). However, it is much slower than at, say 16K context, even when I am not actually using 131K tokens in the first place. Anyone know why this is? I am using the following command on an 8 GB laptop 4060 (I've tried to set aside ~1GB for the OS and whatnot).

llama serve -hf bartowski/Ornith-1.5-9B-GGUF:IQ4_XS --fit on --cache-type-k q8_0 --cache-type-v q4_1 -c 131072 --temp 0.7