OrangePi AI Studio Pro - Qwen3.5-122B-A10B
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
I finally got round to tweaking this, with a bit of help from GLM5.2.
The trick to getting it running with vLLM (which I couldn't get anything really out of before) was when I realized we could write a stub to to implement the rtGetDevMsg to return device capabilities (basically we fake a response from the card) - this is need to get torch_npu running properly on the device.
With that I can finally use vLLM with this, making it actually useful.
I finally got round to tweaking this, with a bit of help from GLM5.2.
The trick to getting it running with vLLM (which I couldn't get anything really out of before) was when I realized we could write a stub to to implement the rtGetDevMsg to return device capabilities (basically we fake a response from the card) - this is need to get torch_npu running properly on the device.
With that I can finally use vLLM with this, making it actually useful.
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.