Qwen-3.8-Next-Flash Ngram Hot-Swappable Knowledge Injector for llama.cpp

Looking into the new Qwen architecture, I was curious if you could modify the Ngram PLE Table to make it work like a long-term knowledge database. It turns out that, with some limitations, you can.

I coded a small modification to llama.cpp to modify the table in-memory, allowing you to patch it with new data in real time. The PLE table is updated on every prompt, so now you can hot-swap parts of it without reloading the model.

The limitation is that it’s hard to control the output reliably, as the embeddings are injected early in the layers. However, with some techniques, you can influence the model’s output with simple modifications, as the example shows.

I created two repos:

1. The modification of llama.cpp here: https://github.com/ortegaalfredo/llama.cpp-NLTM
2. The Ngram knowledge injector (a kind of compiler to create the table patches) here: https://github.com/ortegaalfredo/ngram-knowledge-injector

There are some limitations in the project, as the PLE table needs to be memory-mapped into memory (this is the default in llama.cpp), and I have only tested it with q8 quantization, so you need quite a bit of memory to test this.

Can this be used as a new way of low-cost training? Perhaps. Its not easy at the current state but with simple modifications, I think you could easily create models with long-term instantaneously hot-swappable memory.