NPU
23 updates on NPU.
Vijay Janapa Reddi argues edge generative AI needs models under 1B parameters
In ACM Queue, Vijay Janapa Reddi argues generative AI reaches edge devices only through small deployable models, with sub-1B ones the viable case.
GenAI at the edge survey lists 12 accelerators, 8 of them only simulated
A Johns Hopkins and Duke survey of generative AI on edge devices puts peak accelerator efficiency at 74.34 TOPS/W, with 8 of 12 designs only simulated.
BlueLM-V-3B runs a multimodal model on a Dimensity 9300 NPU in 2.2 GB
CUHK and vivo AI Lab report 24.4 tok/s and 2.2 GB peak memory for a 3B vision-language model on a MediaTek Dimensity 9300.
PhoneLM searches for a fast architecture before training it and hits 58 tok/s
BUPT researchers picked their 0.5B and 1.5B transformer shapes by measuring speed on a Snapdragon 8 Gen 3 first, then pre-training the winner.
AMD trains its first small language model, AMD-Llama-135M, on MI250 accelerators
AMD trained a 135M model from scratch on Instinct MI250 accelerators and reports up to 3.88x faster CodeLlama-7b inference when it drafts tokens.
llm.npu prefills 1,106 tok/s for Qwen1.5-1.8B on a Snapdragon 8 Gen 3
Peking University and BUPT move prompt processing onto the Hexagon NPU and report up to 43.6x faster prefill and up to 59.5x lower energy use.
PowerInfer-2 runs a 47B model on a OnePlus 12 at 11.68 tokens per second
Shanghai Jiao Tong University researchers report a 47B model decoding at 11.68 tokens per second on a OnePlus 12, with weights streamed from flash.
BUPT proposes one 9.2B model in the OS that all apps call through adapters
A BUPT team proposes that the phone OS ship one 9.2B multimodal model all apps call through small adapters, and matched app models on 85% of 50 datasets.
Qualcomm AI Hub opens 75 pre-optimised models and profiling on hosted Snapdragon phones
Qualcomm opened a library of more than 75 models tuned for Snapdragon, with compilation and profiling on real phones in its cloud and two 7B chat models listed.
Snapdragon 8 Gen 3 targets 10-billion-parameter models on device
Qualcomm says the new flagship runs generative models with up to 10 billion parameters on device and reaches up to 20 tokens per second for LLMs.
Qualcomm runs Stable Diffusion on an Android phone for the first time
Qualcomm AI Research generated 512x512 images in under 15 seconds on a Snapdragon 8 Gen 2 phone after quantising the model from FP32 to INT8.