PowerInfer-2 runs a 47B model on a OnePlus 12 at 11.68 tokens per second
Researchers at the IPADS lab of Shanghai Jiao Tong University published PowerInfer-2 on June 10, 2024, an inference framework for running language models that exceed a phone’s memory. On a OnePlus 12, a Snapdragon 8 Gen 3 device with 24 GB of DRAM of which 19 GB can be occupied by an app, the authors measure TurboSparse-Mixtral-47B decoding at 11.68 tokens per second, and describe it as the first system to serve a 47B model on a smartphone. In the same configuration they report 3.12 times the speed of LLMFlash and 21.2 times the speed of llama.cpp. LLMFlash here is their own reimplementation of Apple’s LLM in a flash techniques inside llama.cpp, which they built because that work was never released as code.
PowerInfer-2 splits each feedforward layer into neuron clusters, small groups of neurons that tend to activate together, and treats a cluster rather than a whole matrix as the unit of work. Frequently activated clusters go to the phone’s NPU for dense arithmetic, while rarely activated ones go to the CPU, which copes better with irregular sparse work. Weights that are not already in an in-memory cache are read from UFS flash on demand, and those reads are overlapped with computation so that a core can work on one cluster while the next is still arriving. For prompt processing the system loads whole layers with sequential reads, which the authors measure at 3 times the throughput of random reads, and prefetches the next layer while the current one computes.
The authors vary how much memory the app may use from 7 GB to 19 GB with the 47B model. At 7 GB the cache holds only 1.8 percent of the feedforward weights, so nearly every neuron has to be fetched from flash for every token and decoding falls to 2.13 tokens per second, which is still 1.84 times LLMFlash. The 47B model activates only about 3B parameters per token thanks to its mixture-of-experts structure and added sparsity, which is why it decodes at roughly the pace of a 7B model in the same test. Streaming weights off storage to fit a large model into a small budget is the same bet Edge0 makes at the level of whole experts rather than individual neurons.
On OpenCompass at 4-bit weights, the authors put PowerInfer-2 at an average of 78.38 across Arc-Challenge, Arc-Easy, MMLU and GSM8K for Qwen2 7B against 79.25 for llama.cpp, and at 68.35 against 70.12 for Bamboo-7B, with Qualcomm’s QNN engine at 56.93 and 63.26. Token latency is uneven, and for the 47B model with half the feedforward weights offloaded they report a mean of 99.76 ms per token and a 99th percentile of 140.56 ms, tracking a cache miss rate that averages 3.5 percent but reaches 18.9 percent at the 99th percentile. Prompt processing for Bamboo-7B runs at 404.6 tokens per second with half the weights offloaded, against over 700 tokens per second with everything in memory. They measure energy at 0.257 joules per token against 0.373 for QNN and 0.672 for llama.cpp.
PowerInfer-2 uses privileged system calls such as memory locking, so the authors built and ran it on rooted Android phones, and they state that nothing in the design requires kernel changes and that it should port to iOS. It also needs models with predictable sparsity, which is why the group trained sparsified TurboSparse versions of Mistral and Mixtral, and on an unmodified SiLU-based Mistral-7B the gain over LLMFlash drops to 2.4 times against 4.6 times for the ReLU-based Bamboo-7B. The system is 12,000 lines of C++ added to the group’s PowerInfer engine for PCs, which is MIT-licensed, but the announcement there links the paper and the model weights rather than a release of the PowerInfer-2 code. The paper was last revised in December 2024.