Mobile AI news ticker

Thirty researchers at Huawei Technologies published P/D-Device on August 12, 2025, a scheme that runs a single language model request across a cloud data centre and a phone at the same time instead of choosing one of them. The cloud reads the prompt and writes only the first few tokens, then the device carries the answer to the end. The authors measure average time to first token falling by at least 60 percent against running the whole request on the device, and cloud throughput rising by 1.6 to 15 times, 7.6 times on average, against letting the cloud generate a full 200-token answer.

The split follows from the two phases of inference wanting different hardware. Prefill, where the model reads the prompt before it writes anything, is compute-bound, and the authors report on-device time to first token climbing steeply with prompt length, into seconds or tens of seconds once prompts run to several thousand tokens. Decoding, where the model emits one token at a time, behaves differently, because a phone serves one user while a cloud instance serves a large batch, so the authors find the two roughly matched at tens of milliseconds per output token. Decoding is also what occupies a cloud instance for tens of seconds per request, so the scheme cuts the cloud off after a set number of tokens and hands the rest of the answer to the device.

The device displays the cloud’s first token the moment it arrives, so the wait the user actually perceives is the cloud’s prefill, which the authors put at hundreds of milliseconds. Behind that display the device runs its own prefill while a speed controller releases the next tokens from the cloud at a deliberately slowed pace, matched to human reading speed, until the device has caught up and generates the rest itself. During its prefill the cloud also scores the prompt using the attention weights it has already computed, picks the sentences that matter most, and returns the selection as a compressed mask so the device prefills a shorter prompt, with a quarter of the prompt kept by default.

What crosses the network is text, never model state. The device sends the prompt as raw text and the cloud returns the first token, the mask and a token budget, which the authors cap at hundreds of kilobytes for an 8k prompt, while the KVCache, the gigabyte-scale intermediate state that other cloud-device schemes ship back and forth, is too large to move in time and is never transferred at all. On the prompt text itself the authors state that the request goes out only under user and app authorisation and after sensitive information such as phone numbers and real names has been removed or substituted, and they cite Apple’s Private Cloud Compute as the class of guarantee they have in mind, without describing an implementation or measuring one. The archive’s post on Flower Intelligence covers the reverse arrangement, where the device runs first and the remote service stays off unless the app turns it on.

The prototype runs cloud inference on Huawei Ascend NPUs and device inference on Kirin SoCs, tested on a phone and a tablet, with every chip and model anonymised as Type 1, Type 2, Model 1 and Model 2 throughout the paper. On LongBench with the same model on both sides, the authors’ own table gives their refinement an average of 26.64 on single-document QA against 25.89 for the unrefined on-device baseline, and 26.83 against 22.96 on multi-document QA. The authors name the costs themselves, a quality score around 85 percent of cloud-only inference when the cloud runs the larger model, refinement adding one or two hundred milliseconds at an 8k prompt and several hundred under load, and the smoothed display running slightly above their own tolerable threshold of about a hundred milliseconds per token once prompts pass 8k. The paper, on arXiv under a CC BY 4.0 license, further states that connections were tested over Wi-Fi with an IP whitelist rather than LTE, that collaboration across different NPU types is still being explored, and that the device supports prompts of several thousand tokens against hundreds of thousands in the cloud.

Ten researchers at Beijing University of Posts and Telecommunications, Peng Cheng Laboratory, the University of Cambridge, Xiaomi, Helixon Research and Flower Labs collected 68 small language models between 100M and 5B parameters, released by 24 organisations between OPT in May 2022 and Llama 3.2 in September 2024, and measured both their accuracy and their cost on edge hardware. In Demystifying Small Language Models for Edge Deployment, presented at ACL 2025, Microsoft’s Phi-3 and Phi-3.5-mini sit highest on the capability chart at about 70 percent average accuracy over 10 reasoning and knowledge datasets, level with the Llama 3.1 line the authors draw for comparison. The runtime half of the study covers the 20 models that llama.cpp supports, quantised to 4-bit and run with a 50-token prompt and 50 generated tokens.

Parameter count predicts on-device latency poorly. The measurements put OpenELM-3B at 9.4 percent more parameters than Phi-2 but 9.9 percent faster to the first token, and Qwen1.5-0.5B at 25.4 percent more parameters than Qwen2-0.5B yet 31.9 percent faster to the first token and 11.8 percent faster per decoded token. Qwen1.5-1.8B carries 30.5 percent more parameters than Pythia-1.4B and still reaches the first token 18.5 percent sooner. The authors trace the gap to how a model spends its parameter budget, with Qwen2 sharing its input embedding and output layer and putting the saved weights into wider attention and feed-forward layers, which costs compute in the prefill stage.

Memory ran from 275 MB to 2456 MB across the 20 models, measured with llama.cpp on a Jetson Orin NX at a fixed 2048-token context. Vocabulary size moves that figure as much as parameter count does, and the paper reports Bloom-560M needing 492 MB of compute buffer for its 250,880-token vocabulary, 3.5 times what OpenELM-1.1B needs with 32,000 tokens. OpenELM-1.1B uses 23.8 percent less memory than Bloom-560M despite 32.3 percent more parameters, and OpenELM-3B 13.8 percent less than Gemma-2B despite 21.1 percent more, which the authors put down to the small vocabulary and to grouped-query attention, where several attention heads share one set of cached keys and values. Raising each model to its own maximum context window pushes Qwen2-0.5B past 6 GB, with compute buffer and KV cache making up 85 percent of the total for the Qwen2 models.

Quantisation paid off in decoding rather than in prefill. Testing five settings on Phi-1.5, the authors found Q4_K_M the strongest, roughly halving per-token decode latency against FP16, while at a 50-token prompt it cut the time to the first token by only 13 percent. Q6_K and Q3_K matched or exceeded FP16 on longer prompts, which the paper puts down to the alignment and padding their irregular bit widths force. The hardware gap was wider than the quantisation gap, with the Jetson’s GPU 40 times faster than the Meizu 18 Pro’s CPU at prefill but only 1.84 times faster at decode, and the Xiaomi 12S on a Snapdragon 8 Gen 1+ the fastest of the three phones, ahead of the Pixel 7 Pro and the Meizu on its Snapdragon 888.

The authors name two limits on their own work. They left math datasets out of the capability evaluation because of the gap between small models and larger ones on mathematical reasoning, and they confined the cost analysis to the 20 models llama.cpp supports so that the inference engine stayed constant, excluding every model it cannot load. The paper is free to read at the ACL Anthology under a Creative Commons Attribution 4.0 licence, and the same group had earlier measured 22 models from 0.5B to 7B on four Android phones, three of which reappear in this test bed. PalmBench adds power draw and surface temperature on comparable hardware, and Artificial Analysis has since reported quality, speed and memory together for 33 quantised models on one iPhone 17 Pro.

Apple posted its 2025 foundation models tech report to arXiv on July 17, 2025, giving the numbers behind the roughly 3-billion-parameter on-device model it shipped at WWDC25. The report’s Table 3 puts the price of compressing that model to 2 bits per weight at 3.4 points of MMLU, from 67.8 in 16 bits down to 64.4, with IFEval falling from 85.1 to 82.3.

Apple reaches 2 bits through quantisation-aware training with a learnable scaling factor per weight tensor, and reports that a balanced set of four levels at -1.5, -0.5, 0.5 and 1.5 trains more smoothly than the usual -2, -1, 0 and 1. The embedding table is quantised to 4 bits and the key-value cache to 8 bits, and low-rank adapters are fine-tuned afterwards to claw back the quality the compression costs. The server model takes a different route, compressed after training to 3.56 bits per weight with Adaptive Scalable Texture Compression, a GPU texture format whose fixed-function decode hardware on Apple GPUs unpacks the weights at no cost to the compute cores.

The report also spells out the KV cache sharing the framework announcement only named. Apple splits the transformer so that Block 1 holds 62.5 percent of the layers and Block 2 the remaining 37.5 percent with its key and value projections removed, reusing Block 1’s cache instead. That cuts KV cache memory by 37.5 percent, and because Block 2 produces no keys or values, prefill skips its computation entirely, which Apple says cuts time to first token by about the same 37.5 percent.

Apple rebuilt how the on-device model is trained. The company first trained a dense model on about 14T tokens, sparse-upcycled it into a 64-expert mixture-of-experts version, where only part of the model runs per token, using 1T high-quality tokens, then retrained the dense model on its last 10 percent of tokens against a distillation loss from that MoE teacher. Apple states this cut the cost of training the teacher by 90 percent and removed the need for structural pruning. The server model was trained on 8192 v5p Cloud TPU chips for 13.4T tokens.

For the language expansion the report states the models are designed to support 16 languages, built on a text tokenizer grown from a 100k to a 150k vocabulary and a continued pre-training mix whose multilingual share rose from 8 percent to 30 percent. In Apple’s own human evaluation on US English text prompts, graders preferred the on-device model over Qwen-2.5-3B on 35.3 percent of prompts against 12.9 percent the other way, while against Gemma-3-4B the split was 21.0 percent to 21.9 percent. The report is on arXiv under a Creative Commons Attribution 4.0 license.

Liquid AI released LFM2 on July 10, 2025, three open-weight checkpoints of 350M, 700M and 1.2B parameters aimed at the CPUs inside phones, laptops and vehicles. The company reports 2x faster decode and prefill speed on CPU than Qwen3, and describes the models as built for millisecond latency and offline operation on the device rather than in a data centre.

The architecture is a hybrid of 16 blocks, 10 of them gated short convolutions and 6 grouped query attention. A short convolution mixes each token with a handful of its neighbours instead of attending over the whole history, which is cheap on a phone CPU and keeps the attention cache small, while the six attention blocks preserve the recall that pure convolutions lose. Liquid AI says the layout came out of its STAR architecture search, which it ran against measured peak memory and measured prefill plus decode speed on Qualcomm Snapdragon embedded SoC CPUs rather than against proxy metrics.

Liquid AI benchmarked the models on a Samsung Galaxy S24 Ultra with a Qualcomm Snapdragon SoC and on an AMD Ryzen HX370 laptop, using Q4_0 in llama.cpp and 8da4w in ExecuTorch. In its llama.cpp charts LFM2-350M clears 1,300 tokens per second of prefill and runs at roughly 190 tokens per second of decode on the Galaxy S24 at a 128-token sequence, falling to around 120 tokens per second of decode at 4,096 tokens. The company states that LFM2-700M stays faster than Qwen3-0.6B on both decode and prefill in both runtimes despite being 16 percent larger, and that the CPU gains should carry over to GPUs and NPUs once kernels are optimised.

The models are small enough to ship whole, with Liquid AI’s own Q4_0 GGUF builds at 219 MB for LFM2-350M and 696 MB for LFM2-1.2B. On Liquid AI’s internal evaluation table LFM2-1.2B scores 55.23 on MMLU and 74.89 on IFEval against 59.11 and 73.98 for Qwen3-1.7B, a model with 47 percent more parameters, while LFM2-700M reaches 49.9 on MMLU against 40.08 for Gemma 3 1B IT. Liquid AI notes that it evaluated Qwen3 in non-reasoning mode only, on the grounds that reasoning traces run past the output budgets that matter on edge devices.

Weights are on Hugging Face, with the models also reachable through the Liquid Playground and OpenRouter, and Liquid AI points developers at llama.cpp and ExecuTorch for local testing. The licence is an open licence based on Apache 2.0 that permits academic and research use freely and commercial use by companies under $10 million in revenue, with anyone above that threshold directed to contact the company for a commercial licence. All three checkpoints were trained on 10T tokens, roughly 75 percent English, 20 percent multilingual and 5 percent code, with the context extended to 32K during pre-training.

Update, September 23, 2025. Liquid AI added LFM2-2.6B, which widens the stack to 30 blocks, 22 of them convolutions and 8 attention. The company reports 82.41 percent on GSM8K and 79.56 percent on IFEval, and says the model outperforms Llama 3.2-3B-Instruct, Gemma 3 4B IT and SmolLM3-3B. It ships on Hugging Face under the same licence.

Update, November 28, 2025. Liquid AI published an LFM2 technical report covering the dense 350M to 2.6B models plus an 8.3B mixture-of-experts variant with 1.5B active parameters, where only part of the model runs per token. Measuring on a Samsung Galaxy S25 with a Snapdragon 8 Elite in llama.cpp at Q4_0, it reports LFM2-350M at 1,067 tokens per second of prefill and 194.1 tokens per second of decode on a 1K-token prompt, and LFM2-2.6B at 143 and 33.8 against 57 and 17.2 for Qwen3-4B. The report names ExecuTorch, llama.cpp and vLLM as the shipped deployment packages.

Hugging Face released SmolLM3 on July 8, 2025, a 3B model trained on 11.2 trillion tokens that takes up to 128k tokens of context, covers six languages and answers in either of two modes. It is the successor to SmolLM2, whose largest model was 1.7B with an 8k context window and, in Hugging Face’s own words, primarily understands and generates content in English.

The longer context comes from an architecture change plus a separate training stage. Hugging Face removed rotary position embeddings from every fourth layer, a technique it calls NoPE, and replaced multi-head attention with grouped-query attention to shrink the key-value cache during inference. After the main pretraining run it trained the model on another 100 billion tokens in two steps, 4k to 32k and then 32k to 64k, and YaRN, a method for extrapolating past the trained length, carries it to 128k tokens at inference, though the published configuration ships set to 64k and has to be edited to go further. Multilingual text made up 12 percent of the web data at every pretraining stage, and Hugging Face lists English, French, Spanish, German, Italian and Portuguese as supported.

Hugging Face’s own base-model figures put SmolLM3 at 76.2 on HellaSwag, 65.6 on ARC and 79.0 on BoolQ, ahead of both Llama 3.2 3B and Qwen2.5 3B and ahead of the larger Qwen3 4B on all three. Qwen3 4B stays in front on maths and code, at 51.2 against 46.1 on MATH and 54.9 against 30.5 on HumanEval+, and on the Ruler 64k long-context test Llama 3.2 3B leads at 72.9 against 67.9 for SmolLM3. On the Flores-200 translation benchmark SmolLM3 scores 56.3 against 55.8 for Qwen3 4B and 54.9 for Qwen2.5 3B, and its multilingual chart shows it ahead of both 3B rivals in all five non-English languages and behind Qwen3 4B in all five.

A flag in the system prompt, /think or /no_think, decides whether the instruct model reasons before it answers. Hugging Face reports the reasoning mode at 36.7 percent on the AIME 2025 maths competition against 9.3 percent without it, 41.7 against 35.7 on GPQA Diamond and 30.5 against 15.2 on LiveCodeBench v4. Qwen3 4B leads on most of the eight tests in that comparison in both modes, at 58.8 on AIME 2025 and 52.9 on LiveCodeBench, though SmolLM3 without reasoning tops it on the IFEval instruction-following test at 76.7 to 68.9.

Hugging Face published the recipe as well as the weights. Pretraining ran in three stages that moved web data from 85 percent to 75 and then 63 percent while code rose from 12 percent to 24 and maths from 3 percent to 13, each share drawn from named public datasets, and the company released the training configs with the exact data weights along with intermediate checkpoints. Both the base and instruct models are on Hugging Face under the Apache 2.0 license, and the model card names llama.cpp, ONNX, MLX, MLC and ExecuTorch for local inference. The 4-bit GGUF published by ggml-org on release day is 1.92 GB, against 3.27 GB for the 8-bit build for MLX and 6.15 GB for the bfloat16 weights.

Apple opened its on-device foundation model to third-party apps at WWDC25 on June 9, 2025. The Foundation Models framework gives any app direct access to the roughly 3-billion-parameter model, which runs offline and costs developers nothing per call.

Developers annotate Swift data structures with the @Generable macro, and the framework applies constrained decoding, which Apple calls guided generation, so the model returns those structures rather than free text. Tool calling lets the model invoke functions the app supplies. Developers who need more control can train rank 32 adapters with a Python toolkit.

Apple compressed the model to 2 bits per weight using quantisation-aware training. It splits the transformer into two blocks at a 5:3 depth ratio and shares the key-value cache between them, which Apple says cuts KV cache memory use by 37.5 percent. Images are handled by a 300M-parameter ViTDet-L vision encoder.

The model is designed to support 15 languages. The companion server model on Private Cloud Compute uses what Apple calls a Parallel-Track Mixture-of-Experts design, where independent transformer tracks cut how often the tracks have to synchronise with each other.

Vijay Janapa Reddi of Harvard University published an article in ACM Queue on May 21, 2025, arguing that generative models reach phones, glasses and robots only if the field stops chasing parameter counts and starts designing for deployment. He projects more than 50 billion edge devices by 2030 and writes that today’s frontier models, often hundreds of billions of parameters, cannot leave the data centre. His deployment matrix places only models under 1B parameters in the viable column for true edge hardware, puts medium models of 1B to 10B parameters within reach of high-end phones and single GPUs, and confines anything larger to servers or the cloud.

Reddi grounds that in memory first. He writes that a 6B parameter model at half precision needs roughly 12 GB for weights alone, far above typical mobile RAM, that even a 1B model at about 2 GB can strain some devices, and that small models usually occupy 100 MB to 2 GB. He puts the latency bar at around 50 ms for an augmented reality translation to feel instantaneous and under a second for a dialogue assistant to feel fluid, and he notes that batching across users, which keeps cloud serving efficient, does not apply when requests arrive one at a time. On energy he cites an int8 quantised model using about five times less energy per query than an FP16 one on the same hardware, and says MLPerf Power results suggest model energy efficiency is plateauing even as hardware improves.

Reddi frames the whole problem as a data-model-compute triangle, where scarce local data, limited compute and the need for compact models each worsen the others, and he argues compression alone will not close the gap, since smaller models hallucinate more and can lose their refusal behaviour when quantised hard. He proposes measuring capability and cost together through metrics such as hallucinations per watt-hour, running a model on factual question answering until it consumes 1 Wh and counting the wrong statements, and toxicity per token normalised by model size. The article ran in Queue volume 23, issue 2, and the full text is free to read on queue.acm.org.

Google previewed Gemma 3n on May 20, 2025, a model built for phones, tablets and laptops. Its 5B and 8B versions run with dynamic memory footprints of 2 GB and 3 GB, which Google says matches what 2B and 4B models normally need.

The saving comes from Per-Layer Embeddings, a Google DeepMind technique that keeps embedding parameters off the accelerator. For E2B that cuts what has to be loaded into accelerator memory from 5.44B parameters to 1.91B, with the 2.55B embedding parameters cached to fast storage instead. A MatFormer setup nests a smaller but fully functional 2B model inside the 4B one, so developers can trade quality against footprint. Google reports the model starting to respond about 1.5 times faster on mobile than Gemma 3 4B.

Gemma 3n takes audio, text, images and video as input, including interleaved inputs across modalities, and handles speech recognition and translation. Google states that the same architecture powers the next generation of Gemini Nano, which was to reach Android and Chrome later in 2025. Qualcomm Technologies, MediaTek and Samsung System LSI worked on the optimisation.

Update, June 26, 2025. Google released Gemma 3n as E2B and E4B, naming them for 2B and 4B effective parameters out of 5B and 8B raw ones. Google reports E4B scoring 1303 on LMArena and calls it the first model under 10 billion parameters to pass 1300, ahead of Llama 4 Maverick 17B 128E at 1292 and GPT 4.1-nano at 1288. The release adds a MobileNet-V5-300M vision encoder, which Google says handles up to 60 frames per second on a Pixel, and KV cache sharing that doubles prefill performance. Text covers 140 languages, multimodal understanding 35. Weights are on Hugging Face and Kaggle, with support in Ollama, llama.cpp, MLX, LiteRT and the Google AI Edge Gallery.

Google’s AI Edge team has published Google AI Edge Gallery, an app that runs generative models entirely on the device. The home screen offers three tasks, AI Chat for multi-turn conversation, Ask Image for questions about a photo from the camera or the gallery, and Prompt Lab for single-turn work, which covers free-form prompts, rewriting text in a chosen tone, summarising and generating code. Once a model file is on the phone, the app needs no connection and Google states that all processing happens on the device.

Models come from the LiteRT Community organisation on Hugging Face, which the app’s welcome text links to directly. The first release listed three downloads, Gemma 3 1B IT at 4-bit in 554.7 MB, Hammer 2.1 1.5B at 8-bit and Qwen2.5 1.5B Instruct at 8-bit, the latter two at roughly 1.6 GB each. Gated repositories ask for a Hugging Face access token, which the settings dialog stores and can clear again.

Anything outside that list has to arrive as a LiteRT .task bundle picked from device storage, and the app tells the user that no other file type is supported. Each model then carries a settings dialog with an accelerator switch between GPU and CPU, next to max tokens, top-k, top-p and temperature. The shipped list sets a different default per model, GPU first for Gemma 3 1B and CPU only for Qwen2.5 1.5B Instruct.

After every reply the app prints four of its own measurements, time to first token in seconds, prefill speed and decode speed in tokens per second, and total latency in seconds. It derives them from the session itself, dividing the prompt’s token count by the time the first token took and counting decoded tokens against the clock that starts with that first token, so the numbers describe whatever hardware the app happens to be running on.

Google calls this an “experimental Alpha release” and ships it under the Apache 2.0 licence. There is no store listing at this point, so installing means taking the APK from the project’s releases page on a device running Android 8.0 or newer, with a project wiki covering the corporate-device case. The README lists Android as available now and iOS as coming soon. Underneath, the app builds on LiteRT and Google’s MediaPipe LLM inference API for Android.

Alibaba’s Qwen Team published the Qwen3 Technical Report on May 14, 2025, covering six dense models from 0.6B to 32B parameters and two mixture-of-experts models, where only part of the model is active per token. The two smallest dense sizes, Qwen3-0.6B and Qwen3-1.7B, carry the same mode switch as the rest of the family, so one set of weights either works through a problem step by step before answering or replies straight away. Alibaba reports Qwen3-1.7B scoring 93.4 on MATH-500, a 500-problem maths set, with reasoning on, against 73.0 with it off.

Earlier Qwen releases needed two separate models for that, a chat model and a dedicated reasoning model such as QwQ-32B. In Qwen3 the user picks the behaviour with a flag in the prompt or the system message, /think or /no_think, and the report states that reasoning is the default and that across a multi-turn conversation the model follows the last flag it saw. The same section describes a reasoning budget, where the model’s working is cut off once it passes a token limit the user set and it answers from what it has so far, which the authors say they never trained for and which emerged from merging the two modes.

The benchmark tables give both small sizes in both modes. For Qwen3-0.6B, Alibaba reports 55.6 on MMLU-Redux, a corrected version of the MMLU knowledge test, with reasoning on against 44.6 with it off, and 77.6 against 55.2 on MATH-500. For Qwen3-1.7B the same pair of tables gives 73.9 and 64.4 on MMLU-Redux, 48.3 against 13.4 on the AIME 2024 competition maths set, and 33.2 against 11.6 on LiveCodeBench v5. Against the reasoning baselines in the report, Qwen3-0.6B is ahead of DeepSeek-R1-Distill-Qwen-1.5B on MMLU-Redux, 55.6 to 45.4, and behind it on MATH-500, 77.6 to 83.9.

Alibaba built the small sizes by copying behaviour from the large ones rather than by running the full reinforcement learning pipeline on each. The report describes two stages of distillation from Qwen3-32B and Qwen3-235B-A22B, first on stored teacher outputs generated in both modes, then by matching the small model’s output probabilities against the teacher’s while the small model generates, and it credits the first stage with teaching the smaller models to switch modes at all. Pre-training ran on about 36 trillion tokens covering 119 languages and dialects, up from 29 in Qwen2.5.

Context length is where the two small sizes sit apart from the family. Table 1 of the report puts Qwen3-0.6B and Qwen3-1.7B at 32K tokens against 128K for Qwen3-8B and every larger model, and both tie the input embedding to the output layer so one weight matrix does both jobs. All eight models are published under the Apache 2.0 license on Hugging Face, ModelScope and Kaggle. The Qwen3 blog post of April 29, 2025 names Ollama, LM Studio, MLX, llama.cpp and KTransformers as the tools it recommends for running them locally.

Eight researchers at Google published ML Drift on May 1, 2025, a GPU inference framework for running large generative models on the graphics chip of a phone, a laptop or a Mac. On the Adreno 750 GPU of a Samsung S24, the authors measure 1370 tokens per second of prefill and 37.1 tokens per second of decode for Gemma2 2B, and 412 and 12.7 tokens per second for Llama 3.1 8B. They report a 5 times to 11 times prefill speedup over llama.cpp, MLC LLM and the Qualcomm AI Hub benchmark on Qualcomm GPUs, and the work is due at the CVPR 2025 Workshop on Efficient and On-Device Generation (EDGE).

The team benchmarked five mobile GPUs, the Adreno 830 in a Xiaomi 15 Pro with 16 GB of RAM, the Adreno 750 in a Samsung S24 with 8 GB, the Adreno 740 in a Samsung S23 Ultra with 8 GB, the Arm Immortalis-G720 in a vivo X100 Pro with 16 GB and the Arm Mali-G715 in a Pixel 9 with 12 GB, each at a fixed 1024 prefill and 256 generated tokens. Llama 3.1 8B with every weight at 8-bit ran out of memory on the S24, the S23 Ultra and the Pixel 9 and finished only on the two 16 GB phones, at 7.70 and 4.72 tokens per second of decode. A mixed scheme that holds attention weights at 8-bit and drops embedding and feed-forward weights to 4-bit ran the 8B model on all five devices and lifted decode by up to 1.9 times. ML Drift also treats the two phases of generation separately, since filling the prompt is limited by arithmetic while producing each further token is limited by memory bandwidth.

A mobile GPU constrains inference in ways the paper works through one at a time. Memory comes first, and the authors report that Stable Diffusion 1.4 would need 4.31 GB for its intermediate 16-bit activations, which handing the same buffers to operations whose lifetimes do not overlap cuts to 387 MB. Weight layout comes second, and rearranging weight tensors into slices of four channels that match the 4-element units a GPU computes with buys up to a 20% speedup in matrix multiplication, which the authors call the primary driver of the framework’s advantage. Shader code comes third, and ML Drift writes its kernels once at startup from hand-tuned templates, so the translation between a tensor’s logical indices and the buffer or texture that actually holds it costs nothing while tokens are being produced.

The same engine renders Stable Diffusion 1.4 at 512 by 512 pixels over 20 iterations in 10.96 seconds on the S23 Ultra, under 9 seconds on the S24 and in 3.4 seconds on a laptop with an Intel Lunar Lake Ultra 7 258V. On an NVIDIA RTX 4090 the OpenCL path runs 5% to 25% slower than llama.cpp on CUDA, which the authors put down to NVIDIA’s Tensor Cores being unreachable through OpenCL and WebGPU. The paper names sub-channel quantisation, sparsity and vendor extensions for matrix multiplication as the next steps, and points to no public release of the framework.

Researchers at the Hong Kong University of Science and Technology and Shenzhen University published D2MoE on April 17, 2025, a serving system for mixture-of-experts models on edge hardware, accepted at ACM MobiCom 2025. On a laptop with a 6 GB NVIDIA RTX 3060 and on a Jetson AGX Orin with 64 GB, the authors measure Mixtral 8x7B at 1.14 to 1.39 times the throughput of EdgeMoE and LLaMA-MoE-3.5B at 1.06 to 1.16 times, the second one while using 33 to 53 percent less memory. Against MoQE-DynaIO, which quantises every expert to one fixed bit-width and loads them on demand, they report 1.42 to 3.37 times the throughput.

A mixture of experts replaces the feedforward block in each layer with a set of small networks, and a router sends every token to only a few of them, so the work per token stays small while every expert still has to be held in memory or fetched from somewhere. The paper puts Mixtral 8x7B at over 90 GB in float16, with experts accounting for 89.9 percent of the parameters, against the 6 GB the RTX 3060 laptop offers. One answer is to keep the whole expert set small enough to stay resident, the way Ai2 sizes OLMoE and Meta AI sizes MobileMoE. D2MoE takes the other one, quantising experts hard and loading them from lower tiers on demand, which is also how Edge0 fits a large model into a small budget.

That trade turns a memory problem into a transfer problem. On the RTX 3060 the authors time a single LLaMA-MoE expert at 3.1 ms of computation against roughly 20 ms of data transfer, so the GPU sits idle waiting for weights, and holding several quantised copies of each expert to choose from would give the memory back. D2MoE therefore routes each token twice, once to an expert and once to a bit-width for that expert, and nests the quantised weights so that a wider one contains the narrower ones, which the authors name after matryoshka dolls. They note that 4-bit LLaMA-MoE experts quantised with llama.cpp take 3.81 GB, while storing separate 2-bit, 3-bit and 4-bit copies takes 9.62 GB.

A scheduler then orders expert loads hottest first so that transfers overlap computation inside a set memory budget, and the paper puts the added cost of the second router at 0.28 percent of computation, 0.53 percent of memory and 1.67 percent of latency for LLaMA-MoE-3.5B. On WikiText2 the configuration meant to match 8-bit experts scores 4.09 perplexity on Mixtral 8x7B, against 4.04 for the baseline that holds all 8-bit experts in memory and 4.38 for EdgeMoE. Setting a model up is offline work, at about 2 hours of router fine-tuning plus 10 minutes of quantisation for LLaMA-MoE-3.5B and over 4 hours plus 20 minutes for Mixtral 8x7B. The prototype is roughly 2,500 lines of Python and CUDA on top of PyTorch, written for NVIDIA Ampere and Ada Lovelace, and the paper names no code release and reports no measurements on phones.