PhoneLM searches for a fast architecture before training it and hits 58 tok/s

Researchers at Beijing University of Posts and Telecommunications published PhoneLM on November 7, 2024, a small language model family in 0.5B and 1.5B sizes built in an unusual order. The team first searched transformer configurations for the fastest inference on a specific phone, then pre-trained the configuration that won, instead of choosing a shape for capability and optimising speed after the fact. PhoneLM-1.5B decodes at 58 tok/s on the CPU of a Xiaomi 14 with a Qualcomm Snapdragon 8 Gen 3.

The search varied the layer count from 15 to 25, the ratio of intermediate to hidden size between 2 and 5 and the attention type, and the paper lists measured throughput for seven candidate 1.5B shapes on that phone. The winner has 19 layers and a hidden size of 2,560, a hidden size to layer ratio of 134.7 against the 54.6 to 88.6 the authors observe in recent small models. It uses ReLU rather than the SiLU activation most current small models adopt, which the authors argue is cheaper on NPUs built for integer arithmetic and leaves more zeros in the feed-forward output for sparse-computation tricks. The paper notes that only the 1.5B model went through the search.

Across seven zero-shot tasks including HellaSwag, PIQA, BoolQ and both ARC sets, the benchmark table puts PhoneLM-1.5B at an average of 67.31%, ahead of SmolLM-1.7B at 66.49% and StableLM 2 1.6B at 66.34% and behind DCLM-1.4B at 68.33% and Gemma 2 2B at 71.31%. PhoneLM-0.5B averages 60.43% against 60.17% for SmolLM-360M and 56.46% for OpenELM-450M. The authors measure PhoneLM-1.5B decoding 1.2 times faster than StableLM 2 1.6B and 1.6 times faster than SmolLM 1.7B on the same handset. The two models saw 1.1T and 1.5T training tokens, all of them from open datasets such as DCLM-baseline, StarCoderData and Dolma.

The group ran every device measurement through its own inference engine mllm, with the linear and embedding weights quantised to 4-bit and four threads pinned to the phone’s performance cores. On the NPU it goes through Qualcomm’s QNN framework, where the authors report prompt processing for PhoneLM-1.5B at 654 tok/s on the Xiaomi 14 against 602 tok/s for Qwen 2.5 1.5B. The 4-bit mllm build the group publishes is a 1.41 GB file. The paper states no peak memory figure.

An Android demo app on top of mllm runs PhoneLM-1.5B locally for chat and for Android intent invocation, so a user who asks to be woken at 8 o’clock has an alarm set through the system’s own alarm intent. That variant, PhoneLM-1.5B-Call, was fine-tuned with LoRA on DroidCall, 10,000 synthetic function-calling examples covering alarms, timers, email drafts and searches, and the paper places it above GPT-4o-mini, which scores 71.0% on the same test set. Weights are on Hugging Face under Apache 2.0 and the data preparation, training, fine-tuning and evaluation code is on GitHub under GPL-2.0, alongside the list of open datasets used at each stage.