Claude cli >= 2.1.154 breaks local use with vLLM by introducing "ctx", "msg" and "system" roles for API messages. This 1-line patch to vLLM fixes it.
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
diff --git a/vllm/entrypoints/anthropic/protocol.py b/vllm/entrypoints/anthropic/protocol.py index 3ebc17117..2d5726d73 100644 --- a/vllm/entrypoints/anthropic/protocol.py +++ b/vllm/entrypoints/anthropic/protocol.py @@ -65,7 +65,7 @@ class AnthropicContentBlock(BaseModel): class AnthropicMessage(BaseModel): """Message structure""" - role: Literal["user", "assistant"] + role: Literal["user", "assistant", "ctx", "msg", "system"] content: str | list[AnthropicContentBlock] The changes are (I suspect) related to the new "workflows" feature introduced in 2.1.154. With this patch to vLLM you can use Claude cli workflows with MiniMax-M2.7 (and probably others, this is all I've tested) on vLLM.
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.