Mage-VL is a codec-native, proactive-streaming multimodal foundation model for image and video understanding, whose visual encoder is trained entirely from scratch at a compact 4B scale. It targets a modern Moravec's paradox of VLMs — strong at complex offline reasoning, yet slow and compute-heavy on simple real-time streaming perception. Instead of decoding video into uniformly-sampled frames and pushing a dense grid of patch tokens through a frozen web-pretrained ViT, Mage-VL follows the structure of modern video codecs: it separates a stream into anchor (I) frames and predicted (P) frames, keeps every anchor patch, and retains only the predicted-frame patches where the codec spends bits — the regions carrying real motion and new detail. This codec-aligned sparsity cuts visual tokens by over 75% while preserving spatio-temporal context, yielding up to 3.5× wall-clock inference speedup over uniform frame sampling.

The system pairs two components:

• Mage-ViT — a from-scratch Codec-ViT visual encoder that allocates tokens by codec-derived spatio-temporal importance, on a shared 16×16 patch grid with 3D rotary position encoding. It is codec-agnostic: the same interface accepts a traditional codec (H.264/AVC, HEVC/H.265) via motion vectors + residual energy, or a neural codec (DCVC-RT) via its learned rate map — no architecture or retraining change.
• Qwen3-4B causal decoder — a Qwen3-4B-Instruct-2507 language backbone (the only pretrained component) that consumes Mage-ViT's variable-length token stream through a lightweight two-layer MLP projector, with a unified interface for images, short/long/ultra-long video, and streaming.

On top of this pair, a System 1 & System 2 dual-process design adds proactive streaming inside a single model: a lightweight cognition gate (System 1) watches each rolling codec window and stays silent on routine content, invoking the full VLM (System 2) only when a response-worthy event completes — no multi-agent pipeline required.

✨ Highlights

• Codec-native & from scratch. The entire visual stack is trained from scratch — no billion-scale image-text ViT initialization. The bio-inspired predictive-patch mechanism (I/P frames at 16×16) cuts visual-token consumption by over 75% (~1/8 or less of dense frame sampling), letting the model train on videos 8× longer under the same budget.
• Codec-native speedup. Codec tokenization sets a superior accuracy–efficiency frontier — up to 3.5× wall-clock inference speedup over uniform frame sampling at matched accuracy, and the fastest of all compared models on most video benchmarks (single 8×B200 node).
• Native-resolution scaling. Variable-resolution pretraining lets Mage-ViT improve monotonically with the token budget (peaking >96.1% Food-101 / >86.3% ImageNet at 676 tokens) where fixed-resolution encoders saturate or degrade.
• Matched-LLM video gains. With the 4B Qwen3 backbone held fixed and only the ViT swapped, Mage-VL improves over Qwen3-VL-4B on every reported video and temporal-grounding benchmark — largest on localization-heavy tasks (+22.5 QVHighlight, +17.1 ActivityNet, +11.0 VSI-Bench, +24.5 VideoEval-Pro).
• Strong for its size. On par with Qwen3-VL-4B on static images, and clearly ahead on video understanding and spatial intelligence (+11.0 VSI-Bench, +53.1 CrossPoint, +5.2 EmbSpatial, +22.5 QVHighlight).
• Proactive streaming, single model. A frozen-backbone cognition gate delivers low-latency, event-gated commentary; it tops TimVal / F1 / ROC-AUC / PR-AUC on SoccerNet streaming and generalizes to real 2026 World Cup broadcasts.