https://llmobile.kavents.com/tags/mixture-of-experts/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/edge0-35b-iphone-demo/https://llmobile.kavents.com/ticker/edge0-35b-iphone-demo/Thu, 10 Sep 2026 16:08:02 +0200Samuel Zeng has released Edge0, an open-source framework for streaming mixture-of-experts inference, under Apache-2.0. In his launch post he reports a 35-billion-parameter model running on an iPhone at 1 to 2.5 GB of peak memory without a cloud connection. The repository describes three techniques. Expert weights are streamed from storage on demand, so peak memory is bounded by the active set. A trained prerouter predicts expert routing one step ahead so that loads overlap the forward pass, which the project credits with up to 59 percent higher throughput. Recover-LoRA pairs an int4-quantised frozen base with distilled LoRA adapters. Two tiers are published. edge0-35b has 40 layers and 256 experts, a checkpoint of roughly 23 GB and about 2.9 GiB of peak active memory. edge0-8b has 24 layers and 128 experts, a 4.2 GB checkpoint and about 1.0 GB peak. On a Mac mini M4 Pro with 24 GB, the project measures 14.9 to 17.7 tokens per second of decode for the 35B tier and 23.9 to 25.3 for the 8B tier, with prefill at 113 tokens per second cold and 140 warm for 35B. Quality loss against FP16 baselines is given as 3.9 points on average for the 35B tier and 2.8 points for the 8B tier. The repository states that the MLX backend runs on macOS with Apple Silicon, on M1 through M4, and requires Python 3.10 or newer. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Demo by Samuel Zeng. Watch original on X Source: https://x.com/SamuelZengML/status/2097861839287927139 Read the article: https://llmobile.kavents.com/ticker/edge0-35b-iphone-demo/iPhoneApple SiliconOpen weightsMixture of expertsMemoryhttps://llmobile.kavents.com/ticker/mobilemoe/https://llmobile.kavents.com/ticker/mobilemoe/Tue, 26 May 2026 19:58:00 +0200Meta AI published MobileMoE on May 26, 2026, a family of three language models for phones that hold 1.3B, 2.8B and 5.3B parameters but run only 272M, 528M and 922M of them on any given token. Each one is a sparse mixture of experts, meaning the feedforward block in every layer is split into 60 small networks plus one always-on shared network, and a router picks 4 of the 60 for each token while the rest sit idle. Per-token compute therefore follows the active count, and the memory footprint follows the total. That split is what makes the models awkward to host on a phone. The authors quantise all three to 4 bits with quantisation-aware training, which puts the weights at 0.68 GB, 1.48 GB and 2.75 GB, because every expert has to be reachable even though only a few run per token. On a Samsung Galaxy S25 they measure peak resident memory during inference at 1.49 GB for the smallest model at an 8k context against 1.91 GB for the dense MobileLLM-Pro, and at 4.71 GB for the largest. Sizing the whole expert set to fit phone DRAM is the opposite of the route Edge0 takes, which streams expert weights off storage to keep a 35B model inside a couple of gigabytes. The paper reports that peak memory under real prompts runs 1.2 to 2.1 times higher than under repeated dummy tokens for the mixture-of-experts models, while staying flat for the dense baseline, because varied input routes to more experts and pulls more of them into RAM. Meta AI therefore profiled with real code, knowledge and maths prompts and calls dummy-prompt figures a lower bound rather than a measurement. The authors ran the models through ExecuTorch on the Galaxy S25 with a Snapdragon 8 Elite on 4 CPU threads and on an iPhone 16 Pro with an A18 Pro on 2 CPU threads, after writing a fused mixture-of-experts operator because mobile CPU backends ship none. On a 512-token prompt they measure the smallest model decoding at 130.0 tok/s on the Galaxy S25 against 56.5 for MobileLLM-Pro, and at 180.5 tok/s on the iPhone 16 Pro against 55.1. Time to first token on a 1k prompt drops from 4.26 s to 2.01 s on the Galaxy and from 6.03 s to 2.14 s on the iPhone. Across both phones, CPU and GPU, and the XNNPACK and MLX backends, Meta AI reports the smallest model running 1.8 to 3.8 times faster at prefill and 2.2 to 3.4 times faster at decode than the dense baseline at comparable 4-bit weight memory. Across 14 benchmarks the authors score the instruction-tuned models at 46.7, 55.3 and 60.1 on average at 16-bit precision, against 42.7 for the dense Qwen3.5 0.8B, 44.0 for Gemma 3 1B, 46.9 for Llama 3.2 1B and 50.8 for Qwen3.5 2B, the last of which carries 1.9B active parameters. The largest model reaches 60.1 against 55.6 for Ai2’s OLMoE-1B-7B, with 30 percent fewer active and 23 percent fewer total parameters. After 4-bit quantisation those averages fall to 44.0, 52.5 and 57.8, which leaves the smallest model 1.5 points behind MobileLLM-Pro at 45.5 while scoring 9.4 points higher on MMLU. Meta AI states that the training data across all four stages comes from openly licensed public datasets, names no weight release, and lists mobile NPU deployment, expert pruning and mixed-precision quantisation as further work. Chart: Meta AI. The authors' own comparison of instruction-tuned models at 16-bit precision, with the horizontal axis giving total parameters alongside the 4-bit weight footprint those imply. Source: https://arxiv.org/abs/2605.27358 Read the article: https://llmobile.kavents.com/ticker/mobilemoe/MetaMixture of expertsMemoryQuantisationiPhonehttps://llmobile.kavents.com/ticker/d2moe/https://llmobile.kavents.com/ticker/d2moe/Thu, 17 Apr 2025 07:37:35 +0200Researchers at the Hong Kong University of Science and Technology and Shenzhen University published D2MoE on April 17, 2025, a serving system for mixture-of-experts models on edge hardware, accepted at ACM MobiCom 2025. On a laptop with a 6 GB NVIDIA RTX 3060 and on a Jetson AGX Orin with 64 GB, the authors measure Mixtral 8x7B at 1.14 to 1.39 times the throughput of EdgeMoE and LLaMA-MoE-3.5B at 1.06 to 1.16 times, the second one while using 33 to 53 percent less memory. Against MoQE-DynaIO, which quantises every expert to one fixed bit-width and loads them on demand, they report 1.42 to 3.37 times the throughput. A mixture of experts replaces the feedforward block in each layer with a set of small networks, and a router sends every token to only a few of them, so the work per token stays small while every expert still has to be held in memory or fetched from somewhere. The paper puts Mixtral 8x7B at over 90 GB in float16, with experts accounting for 89.9 percent of the parameters, against the 6 GB the RTX 3060 laptop offers. One answer is to keep the whole expert set small enough to stay resident, the way Ai2 sizes OLMoE and Meta AI sizes MobileMoE. D2MoE takes the other one, quantising experts hard and loading them from lower tiers on demand, which is also how Edge0 fits a large model into a small budget. That trade turns a memory problem into a transfer problem. On the RTX 3060 the authors time a single LLaMA-MoE expert at 3.1 ms of computation against roughly 20 ms of data transfer, so the GPU sits idle waiting for weights, and holding several quantised copies of each expert to choose from would give the memory back. D2MoE therefore routes each token twice, once to an expert and once to a bit-width for that expert, and nests the quantised weights so that a wider one contains the narrower ones, which the authors name after matryoshka dolls. They note that 4-bit LLaMA-MoE experts quantised with llama.cpp take 3.81 GB, while storing separate 2-bit, 3-bit and 4-bit copies takes 9.62 GB. A scheduler then orders expert loads hottest first so that transfers overlap computation inside a set memory budget, and the paper puts the added cost of the second router at 0.28 percent of computation, 0.53 percent of memory and 1.67 percent of latency for LLaMA-MoE-3.5B. On WikiText2 the configuration meant to match 8-bit experts scores 4.09 perplexity on Mixtral 8x7B, against 4.04 for the baseline that holds all 8-bit experts in memory and 4.38 for EdgeMoE. Setting a model up is offline work, at about 2 hours of router fine-tuning plus 10 minutes of quantisation for LLaMA-MoE-3.5B and over 4 hours plus 20 minutes for Mixtral 8x7B. The prototype is roughly 2,500 lines of Python and CUDA on top of PyTorch, written for NVIDIA Ampere and Ada Lovelace, and the paper names no code release and reports no measurements on phones. Chart: Wang et al. The authors' own measurements on the RTX 3060 laptop, where RN is the number of concurrent requests. Each baseline appears as a single point at the budget it needs, while D2MoE is plotted across budgets. Source: https://arxiv.org/abs/2504.15299 Read the article: https://llmobile.kavents.com/ticker/d2moe/Mixture of expertsMemoryQuantisationNvidiaResearchhttps://llmobile.kavents.com/ticker/on-device-collaborative-lm/https://llmobile.kavents.com/ticker/on-device-collaborative-lm/Sat, 21 Sep 2024 00:34:00 +0200Four researchers at EPFL published CoMiGS on September 20, 2024, a scheme in which many devices fine-tune a language model together, with some of the added model parts shared across everyone and the rest kept on the device that trained them. In a simulation of four users, each holding Wikipedia articles in a different language, the authors measure test perplexity at 47.19 with GPT-2 124M as the base model, against 54.38 when each user trains alone on local data and 58.80 for federated averaging, which pools every user’s update into one shared model. Perplexity measures how surprised a model is by the next token, so lower is better. The paper appeared at ICML 2025. Nobody retrains the base model. Each device trains small LoRA adapters on top of pretrained weights that stay frozen, and CoMiGS sorts those adapters into two kinds. Generalist adapters are averaged across all devices at every communication round, specialist adapters and a small router never leave the device, and the router decides per token which adapters to use. The authors train that router against a separate validation set rather than the training data, and report that in the first layer of the network common function words such as “and”, “a” and “the” go mostly to the generalists, while for the users trained on maths and programming text the domain-specific terms in the last layer go mostly to the specialists. Nothing here ran on a phone. The appendix states that every experiment except the centralised baseline ran on a single NVIDIA A100-SXM4-40GB, with four simulated users holding between roughly 400,000 and 1 million training tokens each, drawn from multilingual Wikipedia, SlimPajama, AG News and Common Corpus, and averaged over three seeds. The base models are GPT-2 124M and Llama 3.2 1B. Results in the paper’s table are mixed rather than uniform, with CoMiGS ahead of both baselines on multilingual Wikipedia, SlimPajama and Common Corpus, and behind federated averaging on AG News, at 33.53 against 31.84 with GPT-2 and 16.31 against 15.86 with Llama 3.2 1B, which is the one setup where the test data does not follow the training distribution. The second set of experiments gives users different numbers of adapters, two to four per device with one of them always a generalist, which is how the method is meant to cope with devices of unequal capacity. Against HetLoRA and FlexLoRA, two methods that vary the LoRA rank per device instead, the authors report lower perplexity on every in-distribution task, for example 18.68 against 21.41 and 24.63 on Common Corpus with Llama 3.2 1B. They put the overhead of their extra router at 1.25 percent more computation per forward pass and 1.25 percent more memory, with communication per round halved against federated averaging because only the generalist weights are sent. The paper names its own limits. The authors write that differential privacy on the aggregated generalist weights is something they “do not pursue here”, that robustness against attackers is beyond the scope of the work and that their aggregation carries no guarantee against adversarial parties, and that there is a limit to how much the generalists can prevent overfitting when the local task is easy. They also note that fine-tuning Llama 3.2 1B on multilingual Wikipedia or SlimPajama produced negligible improvements, which they attribute to overlap with its pre-training data, and that is why the Llama runs use Common Corpus instead. The code is on GitHub under the Apache 2.0 licence. Diagram: Fan et al., Figure 2 of the paper, licensed CC BY 4.0. The router and the specialists stay on the device. Source: https://arxiv.org/abs/2409.13931 Read the article: https://llmobile.kavents.com/ticker/on-device-collaborative-lm/Mixture of expertsResearchOpen sourceLlamahttps://llmobile.kavents.com/ticker/olmoe/https://llmobile.kavents.com/ticker/olmoe/Tue, 03 Sep 2024 19:08:20 +0200Ai2 published OLMoE-1B-7B on September 3, 2024, a language model with 6.9B total parameters of which only 1.3B run on any given token. It is a sparse mixture of experts, meaning each layer holds 64 small feedforward networks and a small router picks 8 of them per token, so the rest of the model stays idle. Compute per token lands near that of a dense 1B model, while all 6.9B parameters still have to be held in memory. Ai2 pretrained the model on 5 trillion tokens and reports in the paper that it leads every open model in its active-parameter class. On MMLU the paper puts OLMoE-1B-7B at 54.1, against 48.5 for DCLM-1B, 33.6 for TinyLlama-1B and 32.1 for Ai2’s own dense OLMo-1B. The paper states the model matches or outperforms the dense OLMo-7B at the end of pretraining while using less than half the training FLOPs, and that it beats Llama2-7B on MMLU, scored at 46.2, despite needing roughly 6 to 7 times less compute per forward pass. The instruction-tuned version is reported above Llama2-13B-Chat and DeepSeekMoE-16B. For the training-cost claim the authors ran a controlled comparison, training a 1.3B dense model and a 1.3B active, 6.9B total mixture of experts on 128 H100 GPUs for 130B tokens each. Ai2 reports the sparse model reaching the dense model’s final quality with about 3 times fewer tokens and FLOPs, but only about 2 times faster in wall-clock time, because the larger weight set dropped throughput to 23,600 tokens per second per GPU against 37,500 for the dense run. The full pretraining run used 256 H100 GPUs for around 10 days. Ai2 shipped an iOS app on February 11, 2025 that keeps the whole model on the device, so prompts and responses never reach a server and generation keeps working in airplane mode. Ai2 names the iPhone 15 Pro and 15 Pro Max, the four iPhone 16 models and M-series iPads going back to the 2021 iPad Pro, and states that the first version needs an iOS device with 8 GB of memory. The app runs a 4-bit quantised build of the January 2025 instruction-tuned model, which Ai2 measures at 41 tokens per second on average on an iPhone 16 Pro, on a stack Ai2 built with GenUI on top of llama.cpp. Weights, the training data mix, the training code and the run logs are all published under Apache 2.0, together with 244 intermediate checkpoints. The comparison table in the paper lists OLMoE as the only mixture-of-experts release that opens all four, with most of the other models in the table offering a single checkpoint and no logs. The app is in the App Store and its source code is on GitHub. Screenshot: Ai2. Update, April 14, 2025. Ai2 kept working on the app after the February launch and added macOS support on March 7, 2025 through Mac Catalyst, Apple’s compatibility layer that runs the same iOS code natively on a Mac, together with a counter that shows how many tokens a reply used. A later version reworked the panel that reports generation metrics, and the most recent release, from April 14, 2025, renamed the app to Ai2 OLMoE. Nothing has shipped in the repository since. Source: https://arxiv.org/abs/2409.02060 Read the article: https://llmobile.kavents.com/ticker/olmoe/Ai2Mixture of expertsiOSOpen weightsResearch