llama.cpp
19 updates on llama.cpp.
llama.cpp Hexagon NPU backend tested on a Snapdragon 8 Gen 3 phone
A user report puts Gemma 3 4B at 12.5 tokens per second of generation on a OnePlus 12 using llama.cpp’s Hexagon NPU backend, at about CPU speed but without the heat.
Artificial Analysis benchmarks 33 local models on an iPhone 17 Pro
A benchmark of quantised small models on an iPhone 17 Pro reports intelligence scores, generation times and peak memory between 0.4 and 6.9 GB.
llada.cpp runs a diffusion LLM on a Snapdragon NPU up to 42 times faster
Tsinghua and Beihang researchers report LLaDA-8B generating 128 tokens 17 to 42 times faster on a Hexagon NPU than on the phone CPU.
Tencent open-sources a 440 MB offline translation model for phones
Hy-MT1.5-1.8B-1.25bit compresses a 1.8-billion-parameter translation model from 3.3 GB to 440 MB and runs offline on a phone. Weights and an Android demo are public.
Airgap is a React Native kit for support chatbots that answer offline
Xavier Puspus published a React Native kit for support chatbots that answer without a network, running a 2.4 GB Gemma 4 E2B file through llama.rn.
Show HN: Off Grid runs text, image, vision and speech models offline on phones
An MIT-licensed Android and iOS app bundles llama.cpp, Stable Diffusion and Whisper for offline use. The Show HN thread reached 124 points and 66 comments.
Benchmark of 68 small language models finds architecture outweighs size on device
A study of 68 models from 100M to 5B puts Phi-3 near 70 percent accuracy and finds first-token time and memory tracking architecture, not parameter count.
Liquid AI releases LFM2, three CPU-first models from 350M to 1.2B
Liquid AI released open-weight models of 350M, 700M and 1.2B parameters and reports 2x faster decode and prefill than Qwen3 on CPU.
Hugging Face's SmolLM3 is a 3B model with 128k context and two reasoning modes
Hugging Face released a 3B model with a 128k context window, six languages and a switchable reasoning mode, along with the full training recipe.
Raspberry Pi 5 beats human reading speed only up to Phi 3.5 mini
Helsinki and EURECOM researchers measured 11 models on a Raspberry Pi 5 and a Jetson Orin Nano, where memory and battery gave out before speed did.
Hugging Face trains SmolLM2 at 135M, 360M and 1.7B on up to 11T tokens
Hugging Face released SmolLM2 in three sizes trained on up to 11 trillion tokens, with 4-bit builds from 118 MB for on-device runtimes.
BUPT measures 22 LLMs on four Android phones at about 200 ms per token
A BUPT team ran 22 models from 0.5B to 7B on four Android phones with llama.cpp and found 7B models take about 4 GB and 200 ms per token.