r/LocalLLaMA · · 1 min read

IFM/K2-Horizon-MoVA-36B-A4B-GGUF · Hugging Face

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

IFM/K2-Horizon-MoVA-36B-A4B-GGUF · Hugging Face

more sizes (probably still uploading):

https://huggingface.co/IFM/K2-Horizon-32B-GGUF

https://huggingface.co/IFM/K2-Horizon-7B-GGUF

https://huggingface.co/IFM/K2-Horizon-3.7B-GGUF

https://huggingface.co/IFM/K2-Horizon-0.9B-GGUF

from IFM:

K2-Horizon-MoVA-36B-A4B is the sparse member of the K2-Horizon family: a Mixture-of-Experts model with Mixture-of-Values attention (MoVA) that stores 36B parameters and runs 4B per token. We have released the final checkpoint; intermediate checkpoints, along with the data and the training code, will be released.

K2-Horizon-MoVA-36B-A4B Highlights

  • Frontier-class results at 4B active parameters. On agentic and reasoning benchmarks it outscores open weight dense (approximately 30B model size) and MoE models up to 15× its size; and also performs competitively against closed frontier models (see Benchmark Results).
  • 512K context. Native 524,288-token context from the midtraining stages onward.
  • Intermediate checkpoints. Intermediate checkpoints will be released so capability changes can be studied across training rather than at a single checkpoint.
  • Fully open. Training data/recipe and the training code will be made public.

collection: https://huggingface.co/collections/IFM/k2-horizon

submitted by /u/jacek2023
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA