Ai2 releases OLMoE, 7B parameters with 1B active per token

Ai2 published OLMoE-1B-7B on September 3, 2024, a language model with 6.9B total parameters of which only 1.3B run on any given token. It is a sparse mixture of experts, meaning each layer holds 64 small feedforward networks and a small router picks 8 of them per token, so the rest of the model stays idle. Compute per token lands near that of a dense 1B model, while all 6.9B parameters still have to be held in memory.

Ai2 pretrained the model on 5 trillion tokens and reports in the paper that it leads every open model in its active-parameter class. On MMLU the paper puts OLMoE-1B-7B at 54.1, against 48.5 for DCLM-1B, 33.6 for TinyLlama-1B and 32.1 for Ai2’s own dense OLMo-1B. The paper states the model matches or outperforms the dense OLMo-7B at the end of pretraining while using less than half the training FLOPs, and that it beats Llama2-7B on MMLU, scored at 46.2, despite needing roughly 6 to 7 times less compute per forward pass. The instruction-tuned version is reported above Llama2-13B-Chat and DeepSeekMoE-16B.

For the training-cost claim the authors ran a controlled comparison, training a 1.3B dense model and a 1.3B active, 6.9B total mixture of experts on 128 H100 GPUs for 130B tokens each. Ai2 reports the sparse model reaching the dense model’s final quality with about 3 times fewer tokens and FLOPs, but only about 2 times faster in wall-clock time, because the larger weight set dropped throughput to 23,600 tokens per second per GPU against 37,500 for the dense run. The full pretraining run used 256 H100 GPUs for around 10 days.

Ai2 shipped an iOS app on February 11, 2025 that keeps the whole model on the device, so prompts and responses never reach a server and generation keeps working in airplane mode. Ai2 names the iPhone 15 Pro and 15 Pro Max, the four iPhone 16 models and M-series iPads going back to the 2021 iPad Pro, and states that the first version needs an iOS device with 8 GB of memory. The app runs a 4-bit quantised build of the January 2025 instruction-tuned model, which Ai2 measures at 41 tokens per second on average on an iPhone 16 Pro, on a stack Ai2 built with GenUI on top of llama.cpp.

Weights, the training data mix, the training code and the run logs are all published under Apache 2.0, together with 244 intermediate checkpoints. The comparison table in the paper lists OLMoE as the only mixture-of-experts release that opens all four, with most of the other models in the table offering a single checkpoint and no logs. The app is in the App Store and its source code is on GitHub.

Update, April 14, 2025. Ai2 kept working on the app after the February launch and added macOS support on March 7, 2025 through Mac Catalyst, Apple’s compatibility layer that runs the same iOS code natively on a Mac, together with a counter that shows how many tokens a reply used. A later version reworked the panel that reports generation metrics, and the most recent release, from April 14, 2025, renamed the app to Ai2 OLMoE. Nothing has shipped in the repository since.