On-device
AMD trains its first small language model, AMD-Llama-135M, on MI250 accelerators
AMD trained a 135M model from scratch on Instinct MI250 accelerators and reports up to 3.88x faster CodeLlama-7b inference when it drafts tokens.
Meta ships Llama Stack with Swift and Kotlin clients for on-device inference
Meta standardised its Llama APIs so mobile apps can swap a cloud endpoint for an on-device ExecuTorch runtime without changing the calling code.
Meta releases Llama 3.2 1B and 3B for phones and edge devices
The two lightweight models carry a 128K context window, were pruned and distilled from Llama 3.1, and shipped with day-one Qualcomm and MediaTek support.
CoMiGS splits on-device fine-tuning into shared generalists and private specialists
EPFL researchers split LoRA adapters into shared generalists and device-local specialists, and report 47.19 test perplexity against 54.38 for training alone.
ElastiLM resizes a shared phone LLM per request and switches in 0.31 seconds
Peking University and BUPT built an on-device LLM service that fits model and prompt to each request, switching sub-models in 0.31 seconds.
Ai2 releases OLMoE, 7B parameters with 1B active per token
Ai2's mixture-of-experts model holds 6.9B parameters but runs 1.3B per token, and its iOS app runs it offline on an iPhone 15 Pro or newer.
Gemma 2 2B is distilled from a larger model and scores 1126 on Chatbot Arena
Google released a 2.6B-parameter Gemma 2 trained by distilling a larger model, and reported an Elo of 1126 on the LMSYS Chatbot Arena.
torchchat runs Llama 3 8B on a Galaxy S23 and iPhone at more than 8 tok/s
PyTorch's demonstration codebase exports Llama models to ExecuTorch and ships iOS and Android sample apps for running them on a phone.
Spectra ships ternary 3.9B models that match 4-bit quantisation at half the bits
Nolano AI trained 54 models from 99M to 3.9B parameters on the same 300B tokens in ternary, quantised and half-precision form to compare them by bit size.
Hugging Face releases SmolLM at 135M, 360M and 1.7B parameters
Three base models trained on the newly released SmolLM-Corpus, with published memory footprints from 109.78 MB to 3422.76 MB.
Alibaba builds Qwen2's 0.5B and 1.5B sizes for phones, earphones and glasses
Alibaba built Qwen2-0.5B and Qwen2-1.5B for smartphones, earphones and smart glasses, with 32K context and Apache 2.0 weights.
llm.npu prefills 1,106 tok/s for Qwen1.5-1.8B on a Snapdragon 8 Gen 3
Peking University and BUPT move prompt processing onto the Hexagon NPU and report up to 43.6x faster prefill and up to 59.5x lower energy use.