Toy project: a chat title model that fits in 5 MiB of ram
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Not even sure if I'm allowed to post this, what with the "completely/primarily LLM generated copy" rule (the post itself is fine, but the repo/model I'm sharing definitely is, whoops) and the whole limit self-promotion thing, but it's just a toy I made that is trying to solve a niche I haven't seen much models tackle. I don't really want to put more time into it, but hey, maybe someone will find it useful, and I like open source, so here it is. I'm usually just a lurker in this sub :)
To be honest I coded basically nothing, an LLM wrote almost all of it while I nodded along. It works tho lmao. (I'm actually a programmer, this project just wasn't worth getting into too deeply. I still learned a few surface-level things about how these models work, so that's neat)
TinyTitle is a tiny model (~1.8M params) that turns a chat message into a short title. It's just a small neural net (a GRU thing) that reads your message and either makes up a word or copies one from what you wrote. The whole thing (model + tokenizer + runtime) runs in under 5 MiB of ram, in a few tens of ms (on my desktop), on one small C binary.
Here's a comparison with a bigger reference model (SupraLabs 50M Q8_0), on the same prompts:
| prompt | TinyTitle | Supra Title 50M |
|---|---|---|
| How does AI work? | AI Work | AI Basics Explained |
| How to make a discord server? | Discord Server | Discord Server Creation |
| What's the best way to learn French quickly? | Best Way to French Quickly | Learning French Tips |
| Explain quantum computing like I'm five | Quantum Computing Like | Quantum Computing Basics |
| Can you explain the difference between TCP and UDP? | TCP and UDP Differences | TCP Vs UDP Comparison |
| 36 liters of diesel fuel is worth €18. The tank of this pickup truck can hold 80 liters. How much does it cost to fill the tank? | Diesel Cost Calculation | Diesel Fuel Tank Cost |
These are some decent results, and in general, the Supra model is the best of both (which makes sens, it's bigger). The 50M model is more abstract and grammatical, the tiny one is more literal.
Let's compare the ram usage, including the runtime (and let's use the most agressive quantization of the 50M model (Q1_0)):
| model | file | peak rss |
|---|---|---|
| TinyTitle | 1.98 MB | 4.89 MiB |
| Supra Title 50M (Q1_0, llama.cpp) | 19.6 MB | ~126 MiB |
So about 25x times less ram usage :)
I don't know if people would even want it on Hugging Face for some reason, but if there's enough demand I might put it there and edit this post to add the link. edit: https://huggingface.co/azomDev/TinyTitle
For all I know this is trivial and everyone here already built one, but it was neat to see it work. Thanks for reading, sorry if this is not the right place for this lol.
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.