jinfer: An open-source AI inference engine for the JVM. Finally, AI in jar.

For years, the JVM has watched the AI revolution from the bench. Every model, AI framework, every breakthrough, built with/for Python.

jinfer is an inference engine built for the JVM from first principles: chat, vision, audio transcription, embeddings, reranking, and TTS. No Python runtime, no ONNX, no sidecar process, no wrappers; the whole stack is built for the JVM:

• jinfer Inference engine for the JVM, supports a wide range of popular models and modalities.
• Tok'n'Roll (toknroll) Fast tokenizers for LLMs, pure Java, zero dependencies
• gguf / safetensors native read/write for both major model formats
• jam Quantized matrix multiplication routines (Vector API + optional native backend), competitive with llama.cpp on CPUs
• jota Tensor API targeting Java, C, CUDA, HIP, Metal, OpenCL, and Mojo

It integrates with Spring AI and LangChain4j, and has first-class support for GraalVM Native Image.

Where things stand: this is an early release. CPU is the main target today, and is already competitive with llama.cpp. GPU support via jota is in progress.

Runnable examples + benchmarks: https://qxotic.ai

Jinfer (Apache 2.0): https://github.com/qxoticai/qxotic/tree/main/jinfer

PS: I'm behind it and also the author of llama3.java (2024) and gemma4.java