Applying Sliding Window Attention to pretrained LLMs at inference time [P]

I've been working on a practical implementation of Sliding Window Attention (SWA) for pretrained Hugging Face causal LLMs.

The idea is simple: instead of allowing every generated token to attend to the complete historical KV cache, maintain a bounded cache consisting of:

attention sinks + recent sliding window

I implemented this as a reusable inference layer rather than modifying or retraining the model.

GitHub:
https://github.com/oraby8/SWA

The implementation currently includes:

• bounded KV cache
• circular/ring-buffer storage
• attention sinks
• streaming prefill
• chunked attention masking
• autoregressive decoding
• Full Attention vs SWA benchmarking
• TTFT / TPOT / throughput measurements
• KV-cache memory measurements

One interesting result from my Qwen2.5-7B experiment:

At 16K, SWA-64 also reduced TPOT from ~38.4 ms to ~30.5 ms in this setup.

However, there is an important trade-off: tasks requiring information far outside the active window can degrade. I'm currently investigating how much of this is inherent to SWA versus implementation/model-specific behavior.

I'm sharing the implementation mainly to get feedback from people working on LLM inference, KV-cache optimization, and long-context models.

I'd be particularly interested in:

1. Which model architectures should I validate next?
2. What failure cases should I benchmark?
3. What would make this useful for existing HF inference workflows?
4. Are there cache/attention implementation details I may be overlooking?

Feedback and experiments are very welcome.