MobiLlama is a fully transparent 0.5B model that runs in 770 MB on a phone

Researchers at MBZUAI published MobiLlama on February 26, 2024, a language model of 0.5B parameters aimed at devices that cannot host a large one. The stated aim is a fully transparent small model rather than open weights alone, so the authors release the complete pre-training data pipeline, the training and evaluation code, the weights and more than 300 intermediate checkpoints. They also derive a 0.8B version from the same design.

The design point is a single feed-forward block shared by every transformer layer. Each transformer block normally carries its own feed-forward network, the part that reworks each token’s representation after attention, and the authors measure those layers as 65% of all trainable parameters in their 1.2B reference model. Reusing one block across all 22 layers cuts trainable parameters by 60%, from 1.2B to 0.52B, while keeping the 22 layers and the hidden size of 2048 that the reference model has. The authors report pre-training the shared version in 7 days and 26.6K A100 GPU hours, against 12 days and 46.1K hours for the unshared 1.2B model.

For deployment the authors measured a smartphone with a Snapdragon 685, a laptop with an i7 CPU and a desktop with an RTX 2080 Ti, using 4-bit GGUF builds on the phone and the laptop. On the phone they put MobiLlama 0.5B at 770 MB of RAM and 7.02 tok/s, drawing 5.32 mAh per 1,000 tokens at 13.02% CPU utilisation. They measured Llama 2 7B on the same phone at 4287 MB, 1.19 tok/s and 10.07 mAh per 1,000 tokens with 77.41% CPU utilisation, and Phi-2 2.7B at 1893 MB and 2.88 tok/s. On the laptop CPU the 0.5B model reaches 36.32 tok/s in 799 MB.

MobiLlama 0.5B averages 46.00 across nine benchmarks from the Open LLM Leaderboard in the authors’ own evaluation, which they compare with 43.57 for pythia-410m, and the 0.8B version, made by widening the shared block, averages 46.67. The models were pre-trained on 1.2T tokens from the Amber dataset of LLM360, where the two largest parts are a curated web crawl and code. Weights and chat-tuned variants are on Hugging Face under Apache 2.0, and the repository carries the training code and an Android app package for running the model on a phone.