Android
51 updates on Android.
ShadowNPU scores attention on the Snapdragon NPU and reports 4.5 times faster inference
Peking University and BUPT move attention token scoring onto a Snapdragon NPU and report up to 4.5 times faster inference using a single CPU core.
Benchmark of 68 small language models finds architecture outweighs size on device
A study of 68 models from 100M to 5B puts Phi-3 near 70 percent accuracy and finds first-token time and memory tracking architecture, not parameter count.
Gemma 3n runs 5B and 8B models in 2 GB and 3 GB of memory
Google previewed a mobile-first model whose Per-Layer Embeddings cut RAM use, and said the same architecture powers the next Gemini Nano.
Google AI Edge Gallery runs Gemma 3 1B and Qwen2.5 offline on Android
An experimental Google app downloads LiteRT models from Hugging Face, runs chat, image questions and prompt tests offline, and prints decode speed per reply.
Google's ML Drift runs Llama 3.1 8B on a phone GPU at 12.7 tokens per second
The paper measures 37.1 decode tokens per second for Gemma2 2B and 12.7 for Llama 3.1 8B on the Adreno 750 GPU of a Samsung S24.
Alibaba MNN runs 4-bit LLMs on phone CPUs and GPUs, with a multimodal Android app
MNN-LLM converts PyTorch checkpoints into a 4-bit MNN format for phones, and Alibaba reports prefill 8.6 times faster than llama.cpp on an Android CPU.
EXO Labs benchmarks put Llama 3.1 8B at 14 tok/s on an iPhone 15 Pro
EXO Labs published automated inference benchmarks from real devices, covering iPhone 15 Pro, Galaxy S24 Ultra, Mac mini M4 Pro clusters and desktop GPUs.
PhoneLM searches for a fast architecture before training it and hits 58 tok/s
BUPT researchers picked their 0.5B and 1.5B transformer shapes by measuring speed on a Snapdragon 8 Gen 3 first, then pre-training the winner.
PalmBench finds iPhones running local LLMs about three times faster than Android phones
A benchmark of quantised LLMs on eight phones and boards reports throughput, memory, power and heat, with iPhones ahead and 4-bit drawing more power than 3-bit.
Meta ships Llama Stack with Swift and Kotlin clients for on-device inference
Meta standardised its Llama APIs so mobile apps can swap a cloud endpoint for an on-device ExecuTorch runtime without changing the calling code.
ElastiLM resizes a shared phone LLM per request and switches in 0.31 seconds
Peking University and BUPT built an on-device LLM service that fits model and prompt to each request, switching sub-models in 0.31 seconds.
torchchat runs Llama 3 8B on a Galaxy S23 and iPhone at more than 8 tok/s
PyTorch's demonstration codebase exports Llama models to ExecuTorch and ships iOS and Android sample apps for running them on a phone.