 | I find this new model at HF: "Built for demanding work. A 262 144-token context window, adjustable reasoning effort, tool calling, and text, image and video understanding. Architecture Agnes-3.0-Flash is a hybrid-attention decoder: three of every four layers run a gated delta rule (recurrent, with per-layer state independent of sequence length), and the fourth runs standard global attention. Only 18 of the 72 layers therefore hold a KV cache that grows with context." | Context length | 262 144 tokens | | Decoder layers | 72 = 54 delta-rule recurrent + 18 global attention, alternating 3 : 1 | | Hidden size | 5120 | | Global attention | 24 query heads / 4 KV heads (6 : 1 GQA), head dim 256; RMS-norm on q and k, sigmoid-gated output | | Delta-rule layers | 16 key heads / 48 value heads, head dim 128; causal conv (kernel 4) in front, gated RMS-norm; recurrent state in fp32 | | Feed-forward | SwiGLU, intermediate size 17408; plus a parallel SwiGLU 2048 branch in every layer | | Positions | 3-axis rotary (text / height / width), interleaved mrope sections 11 : 11 : 10, base 1e7, applied to the first 25 % of each head dim (64 dims) | | Vocabulary | 248 320 | | Vision tower | 27 layers, hidden 1152, patch 16, 2 × 2 spatial merge, projected to 5120Architecture Agnes-3.0-Flash is a hybrid-attention decoder: three of every four layers run a gated delta rule (recurrent, with per-layer state independent of sequence length), and the fourth runs standard global attention. Only 18 of the 72 layers therefore hold a KV cache that grows with context. Context length 262 144 tokensDecoder layers 72 = 54 delta-rule recurrent + 18 global attention, alternating 3 : 1Hidden size 5120Global attention 24 query heads / 4 KV heads (6 : 1 GQA), head dim 256; RMS-norm on q and k, sigmoid-gated outputDelta-rule layers 16 key heads / 48 value heads, head dim 128; causal conv (kernel 4) in front, gated RMS-norm; recurrent state in fp32Feed-forward SwiGLU, intermediate size 17408; plus a parallel SwiGLU 2048 branch in every layerPositions 3-axis rotary (text / height / width), interleaved mrope sections 11 : 11 : 10, base 1e7, applied to the first 25 % of each head dim (64 dims)Vocabulary 248 320Vision tower 27 layers, hidden 1152, patch 16, 2 × 2 spatial merge, projected to 5120 | AA shows its 'Proprietary model' but its open: https://artificialanalysis.ai/models/agnes-3-0-flash
[link] [comments] |
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.