Multilingual Tiny (3.7B) Reasoning MoE pretrained from scratch on a consumer-grade GPU

Hello!

I've just uploaded a recent checkpoint of my model trained from scratch:

https://huggingface.co/piotr-ai/polanka_3.7b_exp_wip_260901

It was pre-trained, mid-trained, and fine-tuned on a single 4090 over many months. How many tokens? I lost count.

Feel free to use it as a research artefact.

13 languages: PL, EN, ZH, CS, SK, UK, RU, IT, ES, FR, DE, PT, LT — with extra upscaled data for PL/EN/ZH.