A few months ago, I had the idea of making a LLM from scratch as a personal project (for learning and partly for improving my resume).

Since I learned a lot from other posts on here over the past year, I wanted to share the results.

TLDR: I trained a 1.1B param model on 20B tokens from fineweb-edu, then finetuned it on openhermes with LoRA to get a chat model. Total cost was about $200 (in February/March though, so it would probably be more expensive now).

• code: https://github.com/Ni-co-la-s/gemmeh
• base model safetensors: https://huggingface.co/ni-co-la-s/gemmeh
• instruction-tuned model safetensors: https://huggingface.co/ni-co-la-s/gemmeh-it
• gguf for base and it model (requires my llama.cpp fork so probably not that useful to you): https://huggingface.co/ni-co-la-s/gemmeh-GGUF and https://huggingface.co/ni-co-la-s/gemmeh-it-GGUF
• demo website: https://gemmeh.com/

The architecture is based on Gemma3 since it was my most used model when I started. There are a few differences: - I have a smaller context length (4096) and because of that I didn't use sliding window attention. - I have a smaller vocabulary (32k, trained the tokenizer with sentencepiece) - I also tweaked some hyperparameters to reach my target parameter count.

For the data, I used fineweb-edu for training the tokenizer and pretraining the model. Then LoRA finetuned the model on openhermes. I purposely tried to find data from 2023 and earlier because I saw this post back then and thought it would be cool to test the model by asking it questions about the "future" (like I did in the gallery images).

As far as the training goes:

Pretraining

For pretraining, I first did training runs on 2B tokens to test the architecture at 3 sizes: 185M, 500M and 1.1B. Then I did a final run of the 1.1B model on 20B training tokens.

I did it on vast.ai and here's the summary:

	185M	500M	1B (on 2B tokens)	1B (on 20B tokens)
Total params	185M	527M	1.1B	1.1B
GPU	3090	5090	H100	H100
Duration	19h	17h	13h	130h
Final val perplexity	19.2	16.0	15.1	10.93

Also I logged in wandb generations from a few fixed prompts every 30M training tokens or so (was probably the most fun part of the project to check the new samples every couple hours to see the improvements) Here are a few examples for the final 1B model.

Input prompt: "Let me tell you a story:"

At 30M tokens seen text Let me tell you a story: a person, you should your child, and the other person who can take the time and the person with its own. If you do not want to give them a bit, you can learn from a student

At 20B tokens seen text Let me tell you a story: I lived in a large city and we were having a little get-together. We all knew each other for years – so much so that I was surprised to learn that we met. It was around this time that one of us decided to become a vegetarian.

Input prompt: "The capital of France is"

At 30M tokens seen text The capital of France is by the other of the Western Europe. The U.S. and the church are the first of Christ in 1937, the other three times of the world.

At 20B tokens seen text The capital of France is Paris and its currency is the Euro. A French person is called a Francais. After the Second World War, the French government decided to introduce a new currency that was pegged to the dollar.

Lora finetuning

To get a chat model, I ran some Lora finetuning on the best 1B model, using Openhermes as a dataset. I also did it on vast.ai, but on a 3060 and over 52 hours. Reached a final validation perplexity of 2.71 (not that it means anything since it is not on the same dataset as the previous values)

Again I did have some regular logging of sample prompts.

Input prompt: "What is gravity"

At 3M tokens seen

text The answer is: Gravity is the force that causes objects on Earth to stay together. At 250M tokens seen

text Gravity is the force that causes objects to fall toward each other.

Input prompt: "Write a short poem about a frog."

At 3M tokens seen

text Write a short poem about a frog. eleph. At 250M tokens seen

```text A frog's heart beating In the dark and damp wood A frog's voice, so soft No one can hear.

It's a call, a croak, a chorus Of frogs in the night's air

The land, the air, the water A place where frogs thrive. ```

Input prompt: "What is the chemical formula for water?"

At 3M tokens seen text heatwaves and water. mangan What is the chemical formula of oxygen? mangan At 250M tokens seen text H2O.

Overall, over training that the model became more and more concise, especially compared to the base that was very yappy. Still, the quality is not very good for the total price. (when comparing to nanochat for example). When I have some more time, I will probably experiment with some full sft instead of LoRA, and maybe some extended datasets.

Side-quests

The post is already pretty long so I will just list quickly some of the other things I tried out: - Because my version had some differences with the original Gemma3 and also because I wanted to understand a bit better how it works, I added the architecture in a fork of llama.cpp. - To test it out, I vibecoded a WearOS app I used to run a Q2_K GGUF version of the 1B model (runs at about 2tok/s on my watch) - I ran a few benchmarks with lm-eval, nothing really interesting to note, it is weaker than Gemma3 1B across the board. - I deployed a demo website on GCP (deploying the model on CPU with the GGUFs) to analyze logprobs of the base model (and compare it with a few other small models) and chat with the instruction-tuned model. I don't know much about frontend so the React was completely vibecoded.

Conclusion

Even if the model is not that good, I learned a lot while doing it and I can only recommend to anyone who wants to better understand LLMs. It has also helped me in my job search process over the past 4 months (whether for getting more interviews or for doing better in ML technical interviews)

Let me know if you have any feedback testing the model or any question!