laya.cpp: Optimized laya near-instant decision making

After seeing u/Nandakishor_ml’s post introducing Laya, I wanted to see how fast it could run in a standalone C++ implementation.

Credit to u/Nandakishor_ml for the architecture, training and open-source release. My contribution is the inference implementation: laya.cpp, built on ggml with custom CUDA kernels.

It supports all three checkpoints—English, multilingual and typed-decisions—with native tokenization, model execution and output formatting. There’s also an HTTP server with a JEV-compatible endpoint. No Python or PyTorch is required for inference.

Some English-model results on an RTX PRO 6000 Blackwell, capped at 450 W:

These are questions per second over a fixed 250-question corpus containing choices, scores and booleans. Each precision has paired Python/C++ timings with alternating execution order. Loading and JSON transport are excluded. The README has the full three-model results.

Most of the optimization came from removing unnecessary conversions and copies, fusing operations while preserving rounding, and improving attention memory access.

The code is MIT-licensed. BF16 currently needs the documented CUDA 13.0/cuBLAS 13.1.0 build profile.

Implemented using Codex Astra.