Llama
22 updates on Llama.
MLPerf Mobile v6.0 adds Llama tests in 1B, 3B and 8B sizes
MLCommons added Llama tests in 1B, 3B and 8B sizes to its mobile benchmark app, reporting token throughput next to the existing vision and image tests.
Quant.npu runs 4-bit LLMs up to 15.1 percent faster on a Qualcomm NPU
Eight authors report a fully static integer quantisation method that runs 4-bit models up to 15.1 percent faster on a Qualcomm SM8650 NPU.
FlexServe runs on-device LLM inference inside TrustZone for 4.41 percent more latency
Shanghai Jiao Tong University researchers kept model weights and prompts inside ARM TrustZone at 4.41 percent more time to first token.
Confidant fine-tunes Phi2-2.7B across a phone and two laptops in 40.1 hours
Zhejiang University researchers split a language model across three everyday devices and fine-tuned Phi2-2.7B on Alpaca in 40.1 hours without moving raw data.
ROMA keeps a 4-bit 3B model in on-chip ROM and reports 31,800 tok/s in synthesis
A synthesised 7 nm accelerator holds a 4-bit 3B Llama in read-only memory and the LoRA adapter in SRAM, with no chip and no FPGA prototype built.
Flower Intelligence runs models on device, with remote handoff off by default
Flower Labs released a preview library that runs Llama 3.2 and SmolLM2 locally via WebLLM or MLX Swift, and calls its remote service only if an app enables it.
EXO Labs benchmarks put Llama 3.1 8B at 14 tok/s on an iPhone 15 Pro
EXO Labs published automated inference benchmarks from real devices, covering iPhone 15 Pro, Galaxy S24 Ultra, Mac mini M4 Pro clusters and desktop GPUs.
AMD trains its first small language model, AMD-Llama-135M, on MI250 accelerators
AMD trained a 135M model from scratch on Instinct MI250 accelerators and reports up to 3.88x faster CodeLlama-7b inference when it drafts tokens.
Meta ships Llama Stack with Swift and Kotlin clients for on-device inference
Meta standardised its Llama APIs so mobile apps can swap a cloud endpoint for an on-device ExecuTorch runtime without changing the calling code.
Meta releases Llama 3.2 1B and 3B for phones and edge devices
The two lightweight models carry a 128K context window, were pruned and distilled from Llama 3.1, and shipped with day-one Qualcomm and MediaTek support.
CoMiGS splits on-device fine-tuning into shared generalists and private specialists
EPFL researchers split LoRA adapters into shared generalists and device-local specialists, and report 47.19 test perplexity against 54.38 for training alone.
ElastiLM resizes a shared phone LLM per request and switches in 0.31 seconds
Peking University and BUPT built an on-device LLM service that fits model and prompt to each request, switching sub-models in 0.31 seconds.