NInfer day0 support for Qwen3.8 27b: ~200 tok/s generation, with tons of engine improvments

Qwen3.8-27B is finally here, and NInfer already has Day-0 support!

Weights:
https://huggingface.co/neroued/Qwen3.8-27B-NInfer

Just update to the latest source and give it a try. On a single RTX 5090, NInfer can still reach around 200 tok/s generation with speculative decoding.

NInfer has also improved a lot since my last post.

Now NInfer supports up to 8 concurrent requests, with a shared paged KV cache pool, each request can still use the full context length.

I aslo implemented ReplaySSM for GDN + speculative decoding, which greatly reduces the recurrent-state memory overhead under concurrency, and this is also something vllm does not yet fully supported.

There have also been many cuda kernel optimizations, as well as usage of PDL to further reduce latency.

Feedback and bug reports are always welcome, I'll try to fix issues as soon as possible!