Raspberry Pi 5 beats human reading speed only up to Phi 3.5 mini
Maximilian Abstreiter, Sasu Tarkoma and Roberto Morabito of the University of Helsinki and EURECOM measured 11 language models from 0.5B to 9.2B parameters on a Raspberry Pi 5 and an NVIDIA Jetson Orin Nano Developer Kit, both with 8 GB of RAM, running every model through llama.cpp at 4-bit. On the Pi, the authors report that only models up to Llama 3.2 3B, or up to Phi 3.5 mini in the faster of the two 4-bit formats they tested, generated text above 5.3 tokens per second, the rate they derive from average English reading speed. The Orin’s GPU stayed above that rate for all 11 models, its CPU cores up to InternLM 2.5 7B in that same format.
Generation speed drops off steeply with model size in the paper’s charts. Qwen 2 0.5B runs at roughly 46 tokens per second on the Orin’s CPU cores and roughly 33 on the Pi, while Phi 3.5 mini at 3.8B parameters sits at about 5 on the Pi, about 10 on the Orin CPU and about 15 on the Orin GPU. The authors also contradict Meta’s claim that Llama 3.2 is optimised for Arm processors, since the advantage they saw for Llama 3.2 1B showed up in GPU inference and not on either CPU.
Memory is where the measurements turn hardest. Peak usage grows close to linearly with model size, reaching about 7.9 GB for Gemma 2 9B on the Pi, and it flattens out on the Orin above Phi 3.5 mini because under 7 GB of the nominal 8 GB is usable once the operating system and background processes take their more than 1 GB. Raising the context window to 1028 tokens pushed the largest model into constant page faults on the CPU and made the Orin crash and restart during GPU inference. First-run load times ran from 5 seconds to over 200 seconds, and for Yi 1.5 6B and up the faster 4-bit format raised load times by 17 to 53 times, because those models get evicted from memory and re-read from the microSD card for every inference.
Scaled to the 18.84 Wh battery of a Samsung Galaxy S24 Ultra at full utilisation, the March version of the paper puts Gemma 2 9B at 193 inference runs on the Pi and 101 on the Orin GPU, against 3,928 and 4,280 runs for Qwen 2 0.5B, with no configuration lasting 7 hours. The setup was built to keep thermal throttling out of the results rather than to observe it, so the authors fitted the Pi with an add-on heatsink after it overheated in early experiments, ran the fans on both boards at full speed and left 30 seconds between runs.
Quantisation cost measurable quality. Perplexity, which tracks how uncertain a model is about its next token, rose against the 16-bit weights for every model and both 4-bit formats and rose most sharply for the smallest models, while the downstream benchmarks showed almost no accuracy drop. The paper is on arXiv and under review at an ACM journal.
Update, November 20, 2025. The authors revised the paper after validating their power measurements against external hardware, which raised the measured consumption. The battery estimate for Gemma 2 9B falls from 193 runs to 75 on the Pi and from 101 to 73 on the Orin GPU, and no configuration now reaches 3.5 hours at full utilisation. The speed, memory and quality results are unchanged.