Frontier model pre-training has converged on mixture of experts (MoE), which is fundamentally changing what limits large-scale AI training. As compute per token falls, communication increasingly determines how efficiently models scale across thousands of GPUs. NVIDIA GB300 NVL72 set a world record for pre-training DeepSeek-V3 671B at 1,648 TFLOPs per GPU, showing how advances across the entire AI platform—from silicon to networking to software—continue to push training performance forward. Every gain in pre-training efficiency means researchers can train larger models, run more experiments, and reach frontier capability faster on the same NVIDIA infrastructure.  

The rapid industry shift toward these MoE architectures is driven by their massive computational efficiency. Unlike dense models, where every token activates every parameter and compute per token rises with the total parameter count, MoE models activate a subset of parameters for each token. For example, DeepSeek-V3 holds 671B parameters but activates only ~37B parameters per token, reaching frontier scale at the per-token cost of a far smaller model.

The tradeoff is communication. Those experts live on other GPUs, so every MoE layer must dispatch each token to its experts and gather the results through an all-to-all communication pattern in both the forward and backward passes. This collective sits in the critical path, making throughput as dependent on communication as on compute. Because it occurs at every layer in every training step, small delays compound until the all-to-all can no longer be hidden behind compute, and adding GPUs no longer increases throughput.

This is why pre-training needs a tightly coupled scale-up domain, where every GPU can communicate with every other GPU over a non-blocking fabric that provides high, uniform bandwidth, low latency, and full bisection bandwidth as the domain grows. Training a model of this size requires more GPUs than a single domain can hold, and many domains must be linked together. This scale-out traffic is lighter and less frequent, but it must still complete within the compute window and remain predictable, so no single slow link governs the step. The challenge is two-tiered, and success is measured in delivered FLOPs, not peak FLOPs.

NVIDIA GB300 NVL72: Built for tightly coupled AI pre-training

A two-tiered communication challenge calls for a system designed around it, not just a faster chip. Compute, scale-up interconnect, scale-out networking, infrastructure processing, and software each carry part of the load. A shortfall in any one caps the whole.  GB300 NVL72 addresses these challenges together, a rack-scale system built through extreme co-design, where silicon, interconnect, networking, and software are engineered as one platform rather than assembled from parts. 

At its core is NVIDIA NVLink, the scale-up fabric that makes 72 NVIDIA Blackwell Ultra GPUs work as one. Fifth-generation NVLink gives each GPU 1.8 TB/s of bandwidth and 130 TB/s of non-blocking, all-to-all bandwidth across the rack, so every GPU reaches every other in a single hop. 

The width is only half of it; the path is the rest. NVLink is memory-semantic: a GPU reads and writes a peer’s HBM directly, as native load and store operations, over a lossless, flow-controlled fabric. The transfer is a hardware memory operation rather than a software send, so nothing in the data path adds latency, and reductions can run inside the switch as the data flows through. 

This is what the per-layer traffic demands: the tensor-parallel all-reduce and the MoE all-to-all stay inside the rack, at full bandwidth and low latency. Beyond the rack, the platform scales out over NVIDIA ConnectX-8 SuperNICs at 800 Gbps per GPU, with NVIDIA Quantum-X800 InfiniBand or NVIDIA Spectrum-X Ethernet, to keep the gradient traffic hidden behind the compute. 

Co-design extends beyond the training fabric into infrastructure services and software. In production AI factories, NVIDIA BlueField data processing units (DPUs) provide an isolated infrastructure-processing domain for virtual networking, storage access, security, telemetry, and lifecycle management, reducing host CPU overhead for large-scale training jobs.

Training software covers both sides of the ecosystem. NVIDIA Megatron Core is built and tuned for these GPUs. NVIDIA has also actively contributed to open source frameworks to ensure TorchTitan and JAX run at full speed on GB300 NVL72 systems.

Exceptional pre-training performance with Megatron Core  

On the DeepSeek-V3 671B model, using 256 GPUs, Megatron Core reaches 1,648 TFLOPs per GPU on GB300 NVL72, compared to 606 TFLOPs per GPU on earlier GB200 NVL72 results, ~3x higher delivered throughput per GPU in a single generation. 

NVIDIA continues to optimize its software for better performance from the overall platform. On a pre-training workload the size of DeepSeek-v3 671B, those gains compound. On the same GB300 NVL72 rack-scale system, performance increased 1.5x in six months, driven by software improvements. This shows that raw performance and training throughput continue to improve well after the silicon ships. 

Leading pre-training frameworks accelerated by NVIDIA  

NVIDIA engineers contribute directly to the open frameworks the AI community relies on, adding optimizations that make them faster on NVIDIA GPUs. Developed collaboratively with the PyTorch and JAX communities, these contributions land continuously and improve performance over time.

TorchTitan is PyTorch’s native training stack, and NVIDIA contributions continuously improve its performance on GB300 NVL72. On DeepSeek-V3 671B, these optimizations compound to deliver approximately 6x higher delivered performance on the same infrastructure.

JAX follows the same arc. Over six months, NVIDIA JAX optimizations have lifted performance improvements by nearly 10x on DeepSeek-V3 671B at 256 GPU scale—all from software optimizations. The latest software version reaches an exceptional performance throughput of 1,025 TFLOPS/GPU, and software optimizations continue to evolve from there.  

Scaling DeepSeek-V3 671B pre-training from 256 to 1,024 GPUs, Megatron Core holds 98.5% of its per-GPU performance while TorchTitan and JAX each hold 97%, so nearly all the added infrastructure turns into added system-level tokens per second throughput. That efficiency is the scale-out fabric doing its job efficiently as racks are scaled out with 800 Gb/s per-GPU networking; gradient traffic stays hidden behind compute, ensuring that adding more GPUs strictly increases total system throughput instead of bogging the network down with communication overhead. 

Setting a world record

Pre-training DeepSeek-v3 671B with 256 GPUs achieves a world record 1,648 TFLOPs per GPU with GB300 NVL72, enabling the same training job to reach the same performance with a fraction of the hardware compared to the previous generation. 

The open frameworks tell the same story. Performance continues to improve on the same platform as the software evolves. These results are not the ceiling—they come from a platform whose hardware, interconnect, and software are designed together and continuously optimized. The record-setting performance of today is only the foundation for even higher performance tomorrow. 

Get started training frontier models on NVIDIA AI infrastructure

• Explore rack-scale architecture: Discover the GB300 NVL72.
• Scale your workloads: Get started with Megatron-Core for frontier model training.
• Optimize performance: Read the NVIDIA CuteDSL fusion kernel blog posts to boost performance using CUDA graphs.

Acknowledgements

Setting a world record on the NVIDIA GB300 NVL72 platform by pre-training the DeepSeek-V3 model reflects the work of many exceptional engineers across the company. We would like to acknowledge the contributions of the following individuals (sorted by last name):

Aidyn Aitzhan, Michael Andersch, Jan Bernloehr, Santosh Bhavani, Ben Cashman, Carlo del Mundo, Ashraf Eassa, Fabio Paes Leme Ferriani, Matt Frank, Abhinav Goel, Vivek Goel, Elfie Guo, Eric Harper, Munira Hussain, Tomasz Jakubek, Masaki Kozuki, George Kurian, Himangshu Lahkar, Guihong Li, Kibibi Moseley, Nitin Nitin, Devin O’Kelly, Christian M. Sarofeen, Priya Sethuraman, Tejash Shah, Franciszek Szarwacki, John Tran, Qiyu Wan, and Cliff Woolley.