Huawei opensouced openPangu-2.0-Pro, 505B-A18B

openPangu-2.0-Pro is an MoE model trained on Ascend. The model has 505B total parameters and 18B activated parameters. Its context length is 512k. The total pretraining data contains 34T tokens. During Post-training, openPangu-2.0-Pro is trained through unified SFT with slow and fast thinking capability, multiple specialist RL traning, on-policy distillation combining multiple RL specialists. More details, please refer to openPangu-2.0 Tech Report.

source: https://ai.gitcode.com/ascend-tribe/openPangu-2.0-Pro