https://llmobile.kavents.com/tags/research/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/online-sdft/https://llmobile.kavents.com/ticker/online-sdft/Tue, 18 Aug 2026 16:57:46 +0200I-Ju Lin and Zhang-Wei Hong published Online-SDFT, a method that keeps fine-tuning a small language model on a phone from the user’s own stream of decisions. On a synthetic notification-routing benchmark with Liquid AI’s LFM2.5-230M, the authors report 70.28% of decisions matching a hidden user preference against 52.78% for the strongest baseline they tested, and cumulative regret of 44.76 against 105.26, over 720 decisions in three paired streams. They state that this comparison ran on Apple MPS rather than on a phone, and that the confidence intervals are nominal across only three streams. Self-distillation here means the model teaches itself. The model picks an action first, the phone later observes what happened, and the same model is then run again with that outcome added to its input. Its better-informed answer becomes the training target for the version that saw only the original context, so no second teacher model, no human label and no reward score is needed. The authors keep the 230M base frozen and train a LoRA adapter of 172,032 parameters across 48 tensors in six attention layers, with the teacher pass running with the adapter switched off so student updates cannot move the teacher. Feeding one interaction at a time into so small an adapter makes it noisy and prone to forgetting, so the authors keep a 64-row buffer of recent lessons and sample each batch from it. Their ablation puts that buffer ahead of the exploration schedule as the load-bearing part, with accuracy dropping from 70.28% to 39.58% and regret climbing from 44.76 to 134.90 once replay is removed, while dropping exploration instead leaves accuracy at 69.44% and regret at 62.00. Diagram: I-Ju Lin and Zhang-Wei Hong. The repository carries an Android proof of concept in which LFM2.5-230M decides and ONNX Runtime Training performs the adapter update on the device, with replay and checkpoints in app-private storage that survive a restart. Exporting the model still happens on a Linux host, and the deployment guide asks for Android 12 or newer on ARM64 with 12 GB of RAM at minimum, 16 GB preferred, and 8 GB of free storage, because the training graph is FP32. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: I-Ju Lin and Zhang-Wei Hong. Watch on the project page The authors call that graph a correctness implementation rather than a claim of practical background training on low-memory phones, and state that they did not measure Android latency, peak memory, battery use or thermal behaviour, and that the benchmark uses synthetic streams rather than real users. The code, the Android app and a Colab notebook are on GitHub under the MIT license. No paper or preprint accompanies the release. Source: https://lin826.github.io/SLM-Online-SDFT/ Read the article: https://llmobile.kavents.com/ticker/online-sdft/ResearchDistillationAndroidLFMOpen sourcehttps://llmobile.kavents.com/ticker/fblayout/https://llmobile.kavents.com/ticker/fblayout/Tue, 07 Jul 2026 12:18:25 +0200Researchers at the University of Macau, the University of Georgia and the Hong Kong University of Science and Technology published FBLayout on July 7, 2026, a system that fine-tunes transformer models on a phone’s own GPU, accepted at ACM MobiSys 2026. On a OnePlus Ace 5 Pro with a Snapdragon 8 Elite and its Adreno 830 GPU, the authors measure training on Llama 3.2 1B, Qwen2.5 1.5B and Gemma 2 2B at 3.9 to 4.1 times the speed of MNN, 4.3 to 4.9 times TFLite and 5.4 to 5.7 times TVM. Over the same three baselines they count 3.5 times fewer global memory reads and 4.2 times fewer cache misses. The authors time inference on the Snapdragon 8 Elite at roughly 6.5 times faster on the GPU than on the CPU, while fine-tuning models of 0.5B to 2B parameters on the GPU comes out level with the CPU or up to 43 percent worse. Their explanation is memory layout, meaning the order in which a tensor’s numbers physically sit in memory. Mobile GPUs hold tensors in a 2D texture grid whose cache rewards reading neighbouring cells, and training reuses forward-pass activations and weights in the backward pass, which sums along a different dimension and so walks the same tensor in strides instead of in rows. Each stride that misses the cache becomes a fetch from DRAM at under 100 GB/s of mobile memory bandwidth, and the GPU stalls waiting for it. Existing frameworks take one of two ways out, and the paper prices both. MNN reuses the forward layout in the backward pass, which the authors put at up to a 5.3 times slowdown on key operations and at under 21 percent arithmetic-unit utilisation in a matrix-multiply benchmark. TFLite instead inserts explicit transpose and reshape operators, and the paper puts that route at 40.8 to 50.0 percent of total runtime across the three LLMs and at roughly a third of all memory allocations. FBLayout defines one tile-based layout, called R-Tile, that keeps both directions contiguous, then rewrites the remaining reshapes as coordinate arithmetic so no data moves, which they report covers over 86 percent of all layout transformations and over 95 percent of those in the LLMs. The evaluation covers seven models, adding BERT-Large, ViT-Large, Whisper-Large and Stable Diffusion 1.5 to the three LLMs, with LoRA adapters in the attention and MLP blocks for everything except ViT. Beyond the Snapdragon 8 Elite phone the authors test a OnePlus Ace 10 Pro with a Snapdragon 8 Gen 1 and an Adreno 730 and a OnePlus Ace 5 Ultra with a Dimensity 9400+ and an Arm Mali Immortalis-G925, and report gains on both. Preparing the Llama 3.2 1B training graph takes 6.1 seconds in their measurement, against 723 seconds in MNN, 862 in TFLite and over 4,800 in TVM, and a Monsoon power monitor puts energy use over five training rounds at 3.5 to 6.3 times below the baselines. FBLayout is built on top of Alibaba’s MNN as about 9.2K lines of C++ and OpenCL, extending it with automatic differentiation so the training graphs exist in the first place. The paper names no code release and points to no repository. It appeared at MobiSys 2026 in Cambridge in June 2026 and is on arXiv under a CC BY-NC-ND 4.0 license. Source: https://arxiv.org/abs/2607.21624 Read the article: https://llmobile.kavents.com/ticker/fblayout/ResearchMemoryQualcommMediaTekAndroidhttps://llmobile.kavents.com/ticker/on-device-diffusion-llm-npu/https://llmobile.kavents.com/ticker/on-device-diffusion-llm-npu/Thu, 11 Jun 2026 14:44:00 +0200Tuowei Wang, Yanfan Sun and Ju Ren of Tsinghua University and Beihang University published llada.cpp on June 11, 2026, an inference framework that runs diffusion language models on a phone’s NPU. On an OnePlus Ace5 Pro with a Snapdragon 8 Elite, the authors measure LLaDA-8B-Instruct generating 128 tokens 38 times faster than the same model on the phone’s CPU for GSM8K prompts and 42 times faster on BoolQ, and they report a range of 17 times to 42 times across their phones and tasks. The same figure puts Llama-3-8B-Instruct on the NPU at 19 to 21 times that CPU baseline, and the authors state that llada.cpp runs LLaDA-8B up to 3.9 times faster than an autoregressive model of the same size. Chart: Wang, Sun and Ren, CC BY 4.0. A diffusion language model does not emit one token after another. It starts from a block of masked positions, 32 tokens by default here, and refines all of them together over repeated denoising steps, committing the positions it is most confident about and carrying the rest into the next step. That turns decoding into a run of large matrix operations, which is the work a mobile NPU is built for, while autoregressive decoding hands the NPU one token’s worth of work per step. The cost the paper identifies is that every denoising step recomputes the full sequence, so the parallel work gets repeated many times over. llada.cpp is built on llama.cpp and targets Qualcomm’s Hexagon NPU with 4-bit weights. Once a block is nearly done only a few masked tokens are left, so the framework pulls tokens from the next block into the same NPU pass to keep it full, keeps already committed tokens open to revision and repairs the unstable ones on the CPU while the NPU carries on, and repacks the limited address space the NPU can see so weights and cache do not have to be remapped between steps. The authors’ breakdown of a 128-token GSM8K request on the Snapdragon 8 Elite puts it at 2996.2 s on the CPU, 607.0 s once the prefix KV cache is reused, and 16.1 s with the full framework. Accuracy moves in both directions. Measured on 200-sample subsets against LLaDA-8B-Instruct on the CPU, the authors’ table has GSM8K rising from 39.0 to 43.5 and ARC-C from 84.0 to 85.0, while BoolQ falls from 82.5 to 80.5 and HellaSwag from 51.0 to 49.5, and their ablation shows the borrowed future-block tokens costing accuracy on all four tasks before the CPU-side revision path recovers it. The work covers three 16 GB phones, an OnePlus 12 with a Snapdragon 8 Gen 3, the Ace5 Pro, and an OnePlus 15 with a Snapdragon 8 Elite Gen 5, and the authors state that on the Snapdragon 8 Gen 3 their framework is still slower than Llama-3-8B at 128 tokens and only pulls ahead on the two newer chips. The paper is on arXiv under CC BY 4.0 and names no code release. Source: https://arxiv.org/abs/2606.13740 Read the article: https://llmobile.kavents.com/ticker/on-device-diffusion-llm-npu/QualcommNPUAndroidllama.cppResearchhttps://llmobile.kavents.com/ticker/caped-privacy-defence/https://llmobile.kavents.com/ticker/caped-privacy-defence/Wed, 10 Jun 2026 22:48:00 +0200Researchers at the Chinese University of Hong Kong, Southeast University and Shandong University published CAPED on June 10, 2026, a filter that blacks out parts of a phone’s screenshots before they reach a cloud-hosted GUI agent. Across a 28-task privacy suite the authors built themselves, weighted seeded leakage falls from 0.766 with raw screenshots to 0.268 with the filter in front, counted only on runs where the agent still finished its task. A multimodal model plays the attacker and scores what it can still infer from the whole screenshot sequence, not from single frames. A screenshot-based agent operates a phone the way a person does, by looking at the screen, so every image it uploads carries whatever happens to sit next to the thing the user asked for. The authors call this incidental visual privacy exposure and name contacts, message previews, non-target photos, filenames, recommendation feeds and health widgets as content the remote model receives although the user never mentioned it. In one of their test shops, a request to buy sunglasses puts prenatal vitamins, a baby bottle set and a maternity dress in front of the model, because the shop’s recommendation rows share the page with the product. Stripping names and numbers out of the text does not close that gap, and the authors measure text-only anonymisation at 0.752 against 0.766 for no protection at all. Figure: Shen et al., licensed CC BY 4.0. The mock SunShop task asks the agent to buy sunglasses, so the personalised rows around it count as incidental exposure. CAPED decides per interface element rather than per screen, according to the paper. It first sorts the current screen into one of three postures, so galleries, maps, files and financial views show their text but hide their images, while messaging, contacts, feeds and health dashboards hide both by default. The user’s instruction is turned into a short list of what the agent is allowed to need, and a hidden element is released only when it matches that list, checked either by text matching or by asking a vision model whether a cropped thumbnail really shows the requested subject. Hidden areas keep their place in the layout under a solid overlay with an eye icon, and names in exposed text are replaced by placeholders the remote agent can still follow. The defence is meant to run on the phone, and the authors argue the instruction has to be interpreted locally too, since it can name people, relationships, locations and intentions before any screen policy applies. The prototype they measured does not run on a phone. They state that CAPED’s components ran on the evaluation host with an evaluation-side service for the heavier models, and that the timings should be read as live-loop prototype latency rather than physical-phone latency, energy or thermal behaviour. Over 1,394 measured steps the privacy work added 2.325 seconds per agent step, 22.6 percent of a 10.835-second step, most of it the detector at 1.811 seconds, plus 5.716 seconds once per task to extract the requirements. The protection also costs task success. On AndroidWorld’s 116 ordinary tasks, GUI-Owl-7B completed 77 of them (66.4%) unprotected and 64 (55.2%) behind the full filter, which the authors call a prototype-level utility cost. They name further limits themselves, writing that screens their rule base does not cover fall back to the most permissive posture, which they describe as not privacy-conservative, and that open-ended requests such as finding the latest receipt widen exposure and account for much of the leakage that remains. On the connected Gmail, Drive and Photos tasks the filter moves leakage from 0.402 to 0.347 only, the interactive unmasking path was never tested with real users, and the authors write that they plan to release the CAPED artifact in a future public version. Source: https://arxiv.org/abs/2606.12666 Read the article: https://llmobile.kavents.com/ticker/caped-privacy-defence/AgentsAndroidPrivacyResearchBenchmarkshttps://llmobile.kavents.com/ticker/quant-npu/https://llmobile.kavents.com/ticker/quant-npu/Tue, 19 May 2026 12:48:00 +0200Eight authors published Quant.npu on May 19, 2026, a way of quantising 1B to 3B language models so they run on a phone’s neural processing unit with no rescaling at runtime. On a Qualcomm SM8650 NPU they measure their 4-bit weight, 8-bit activation build of Llama-3.2-3B-Instruct finishing up to 15.1 percent faster than the same model under ExecuTorch’s 4-bit per-block weights with 16-bit activations, for about 1 percentage point of zero-shot accuracy. Across every model and task in the paper the authors put the average cost at a 2.58 percent accuracy drop and a 1.23 rise in perplexity. Quantisation stores a tensor as small integers plus a scale factor that maps them back to real values. Most published 4-bit methods, including the QuaRot and SpinQuant work Quant.npu is measured against, work that scale out while the model runs by scanning each activation tensor for its largest value. A mobile NPU compiles its graph before the model ships and expects every scale to be fixed by then, because that scan is a reduction the hardware is poor at, at least 7 vector instruction cycles for a 128-element 8-bit vector by the authors’ count, and because two of the four rotation steps these methods use against outliers need floating-point matrix multiplications during inference, which the paper puts at roughly double the compute. Anything left dynamic therefore has to run outside the compiled integer pipeline, and the authors report that simply freezing a model optimised under dynamic quantisation costs 15.61 percent accuracy on SmolLM2-1.7B-Instruct. Quant.npu learns the scales during optimisation instead and folds its two rotation matrices into the weights offline, so the shipped graph is integer arithmetic end to end. In the paper’s main table for Llama-3.2-3B-Instruct at 4-bit weights and 8-bit activations, that lifts average zero-shot accuracy across six benchmarks to 58.27 percent against 46.23 for SpinQuant and 47.66 for QuaRot in the same static setting, with perplexity on C4 falling from 28.78 to 19.16. Against the slower ExecuTorch configuration it beats on latency, though, Quant.npu is the one behind, scoring lower on five of the six benchmarks, 1.08 points lower on average and 0.97 higher in perplexity, and FP32 stays ahead of both at 61.3 percent. On the newer SM8750 NPU the authors report peak memory of 1876 MB to 1985 MB for their 3B builds against 2216 MB to 2435 MB for ExecuTorch, and prefill of 969.72 tok/s for Qwen2.5-3B-Instruct on the DroidTask set against 808.10. They name two limits of their own, that the pipeline still keeps a few operations such as the SiLU activation at 16 bits to hold accuracy, which they call a bottleneck against a fully low-bit pipeline, and that they use no method for picking calibration data even though the optimisation is sensitive to it. Quantising Qwen3-1.7B takes about 2.5 hours on NVIDIA A40 GPUs, and the preprint announces no code release. Source: https://arxiv.org/abs/2605.20295 Read the article: https://llmobile.kavents.com/ticker/quant-npu/QualcommNPUQuantisationLlamaResearchhttps://llmobile.kavents.com/ticker/small-vlms-video-compressors/https://llmobile.kavents.com/ticker/small-vlms-video-compressors/Thu, 09 Apr 2026 13:40:25 +0200Researchers at Meta AI and KAUST published Tempo on April 9, 2026, a way to hand a long video to a small vision-language model first and let a larger language model answer the question afterwards. On LVBench, a benchmark whose clips average 4101 seconds, the authors report their 6B system scoring 52.7 against 48.2 for VideoChat-Flash 7B, the strongest specialised baseline in their table, and against 30.8 for GPT-4o and 33.1 for Gemini 1.5 Pro. The small model does the reading and the large one does the reasoning. A Qwen3-VL-2B-Instruct compressor takes an eight-frame chunk of video together with the user’s question and writes out a short run of memory tokens that hold what the chunk contains, and a Qwen3 4B language model then reads the whole condensed sequence with timestamps attached and produces the answer. A linear projector joins the two, for 6B parameters in total. The compressor is the part that could in principle sit on a phone, since it never sees more than a few seconds of video at a time. How many tokens each chunk gets is decided during the same forward pass. The authors insert a yes or no question about whether the segment is relevant to the query, read the difference between the two logits as a score, and keep more memory tokens for chunks that score high, spending between 0.5 and 16 tokens per frame under a fixed overall budget of 4K or 8K visual tokens. On LVBench under the 4K budget the measured average comes to 2.9 tokens per frame. The tighter budget also scores better than the looser one on that benchmark, 52.7 against 52.3, and on the long subset of Video-MME, 57.8 against 57.0. Raising the cap to 2048 frames and a 12K budget brings LVBench to 53.7. Diagram: Fei et al., Figure 2 of the paper, licensed CC BY 4.0. On the other three benchmarks the authors report 67.8 on Video-MME under the 4K budget against 65.3 for VideoChat-Flash and 61.9 for the Qwen3-VL 2B model the compressor starts from, plus 75.6 on MLVU and 65.1 on LongVideoBench. All of these are accuracy scores run through lmms-eval, not speed or memory figures. Nothing in the paper was measured on a phone, a tablet or any other edge device. Training used 32 and then 64 Nvidia H100 GPUs, the paper reports no latency, throughput or memory footprint for the compressor on mobile hardware, and the on-device case rests on the size of the model rather than on a measurement. The authors state that all training datasets they used are publicly available, and they link code and a project page from the paper, with the repository carrying an Apache 2.0 license. They describe no weights release in the paper itself. Their stated next steps are to let the compressor decide for itself when it has gathered enough evidence to stop, which they say would bottleneck inference latency as things stand. Source: https://arxiv.org/abs/2604.08120 Read the article: https://llmobile.kavents.com/ticker/small-vlms-video-compressors/MetaQwenResearchBenchmarksDistillationhttps://llmobile.kavents.com/ticker/llm-inference-at-the-edge/https://llmobile.kavents.com/ticker/llm-inference-at-the-edge/Tue, 24 Mar 2026 19:28:00 +0100Four authors at Conscious Engines published a benchmark of Qwen 2.5 1.5B at 4-bit across four edge platforms on March 24, 2026, feeding each device the same 258-token prompt 20 times in a row with a 1-second gap instead of once. On an iPhone 16 Pro the first two runs peak at 40.49 tokens per second, and from run 17 onwards the phone holds a plateau of 23.67 tok/s, which the authors put at 41.5 percent below that peak. The iPhone passes through the three thermal states iOS reports to an app. Runs 1 and 2 sit in Normal and average 36.83 tok/s, runs 3 to 16 sit in Warm and average 24.96 tok/s, and runs 17 to 20 sit in Hot at 23.67 tok/s with a variation of 1.2 percent. The authors write that the 1-second gap allows no thermal recovery, and they measure 5 percent of battery drained over the 20 runs, which they project to roughly 400 replies of this length per charge. A Samsung Galaxy S24 Ultra degrades more gently in the same test. Its Adreno 750 GPU boosts to 1,000 MHz on runs 4 and 5 for a peak of 12.21 tok/s, then steps down to a 720 to 770 MHz plateau from run 8 where the authors measure 10.38 tok/s, 15 percent off the peak, with a low of 9.55 tok/s on the final run. They stress that reaching this plateau took deliberate tuning, with the display off, the context capped at 2,048 tokens and MLC-LLM’s prefill chunk cut to 128 tokens, and that earlier unconstrained runs hit an OS-enforced GPU frequency floor. Neither phone ran the model on its NPU. The paper states that MLX dispatches to the A18 Pro GPU through Metal and does not target Apple’s Neural Engine, and that the S24 Ultra ran through MLC-LLM on the Adreno GPU, leaving the Snapdragon 8 Gen 3’s Hexagon NPU outside the measurement. The one NPU that did run the model, a Hailo-10H module in a Raspberry Pi 5, was also the only part that never throttled, holding 6.914 tok/s across all 20 runs with a variation of 0.04 percent at under 2 W, its CPU at 52.7°C and the NPU at 58.5°C, with attention layers that hailo-ollama does not offload falling back to the Pi’s Cortex-A76 CPU. A user running llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone reported 12.5 tok/s of generation on Gemma 3 4B and said the phone stayed cool. A laptop RTX 4050 on battery led on speed at 131.7 tok/s and 34.12 W, its GPU climbing from 55°C to 70°C without throttling. Energy per token comes out at 270.5 mJ for the Hailo module measured across the whole system, 297.3 mJ for the RTX 4050 measured at the GPU, and 143.0 mJ for the S24 Ultra plateau derived from the Android fuel gauge with the display off. The authors state that the three readings differ in scope and call the resemblance suggestive rather than settled, and report no energy figure for the iPhone because iOS exposes no per-component power reading. They tested one model, one prompt and one unit per platform across four different runtimes, and describe the results as characterising hardware and software together rather than hardware on its own. Source: https://arxiv.org/abs/2603.23640 Read the article: https://llmobile.kavents.com/ticker/llm-inference-at-the-edge/BenchmarksNPUiPhoneSamsungResearchhttps://llmobile.kavents.com/ticker/mobilellm-flash/https://llmobile.kavents.com/ticker/mobilellm-flash/Mon, 16 Mar 2026 23:10:00 +0100Meta AI published MobileLLM-Flash on March 16, 2026, three models of 350M, 650M and 1.4B parameters whose shapes were picked by measuring how fast each candidate actually ran on a phone. Meta reports the family reaching up to 1.8 times faster prefill and 1.6 times faster decode than Liquid AI’s LFM2 models on mobile CPUs, at comparable or higher accuracy, with support for up to 8k tokens of context. Meta’s first MobileLLM paper argued in 2024 that deep and thin networks win below a billion parameters, and the Flash paper states that such deep-and-thin structures often fail to improve on-device latency. Meta exported 100 architectures to a phone and found parameter count a poor predictor of measured prefill and decode speed, at a Kendall rank correlation of 0.40, with FLOPs little better at 0.46 and 0.55. The 1.4B model therefore runs 16 layers at a width of 2048, against the 30 layers at 1280 of MobileLLM-Pro-1B. Meta ran the latency tests on a Samsung Galaxy S25, a phone with a Snapdragon 8 Elite chipset, an octa-core CPU and 12 GB of memory, exporting each model through ExecuTorch to the XNNPACK CPU backend at 4-bit weights, 8-bit dynamic activations and a quantised KV cache, on 4 CPU threads and averaged over three runs after a warmup. At a 2k context the 650M model reaches its first token in 3.34 s against 6.01 s for LFM2 700M, and decodes at 85.35 tok/s against 53.57 tok/s. Meta repeated the 1k-context test on an iPhone 17, where the 650M model takes 2.81 s to the first token against 3.40 s for LFM2 700M. Most of the speed comes from letting some layers skip attention, so a token passes through those blocks without being compared to every earlier token. The 350M model keeps full attention in 7 of its 12 layers and the 650M model in 8 of 13, while the 1.4B model attends in all 16. Meta reports that skipping consistently beat sliding window attention, which instead limits each token to a fixed span of recent ones, and that more than three skipped layers in a row costs accuracy at the same latency, dropping TriviaQA exact match from 33.2% to 8.8%. Meta built the three models by pruning a shallow 1.8B variant of MobileLLM-Pro and inheriting its weights, which the paper says took about 35% of the tokens that pretraining MobileLLM-Pro required. Across nine reasoning and knowledge tasks Meta reports averages of 45.46, 48.57 and 55.06 for the three sizes, against 44.92, 48.52 and 50.48 for LFM2 at 350M, 700M and 1.2B, 51.79 for Gemma 3 1B and 49.01 for Llama 3.2 1B. After instruction tuning, the 1.4B model scores 47.89 on MMLU and 46.34 on HumanEval, against 49.30 and 37.80 for Llama 3.2 1B. The paper announces no release of the weights. Chart: Meta. Bubble area shows parameter count, latency measured on a Samsung Galaxy S25. Source: https://arxiv.org/abs/2603.15954 Read the article: https://llmobile.kavents.com/ticker/mobilellm-flash/MetaResearchBenchmarksAndroidPyTorchhttps://llmobile.kavents.com/ticker/flexserve/https://llmobile.kavents.com/ticker/flexserve/Tue, 10 Mar 2026 01:31:25 +0100Researchers at Shanghai Jiao Tong University published FlexServe on March 10, 2026, a serving system that keeps an on-device language model’s weights and the text going through it inside ARM TrustZone’s secure world. Running Llama 3.2 3B, Llama 3.1 8B, Qwen3 1.7B and Qwen3 8B quantised to 8-bit on a NanoPC-T6 board with a Rockchip RK3588, 16 GB of memory and a 6 TOPS NPU, the authors measure time to first token 4.41 percent above the same inference run without any protection, averaged over their prompt lengths. Against a TrustZone baseline that already uses a secure NPU and a loading pipeline, they report FlexServe 1.43 to 2.42 times faster, 1.85 times on average. TrustZone splits the chip into a normal world, where Android and every app runs, and a secure world the normal world cannot read. FlexServe puts the inference engine, the weights, the prompt, the generated text and the KV cache on the secure side, so a kernel an attacker has taken over cannot read any of it out of memory, and the authors justify that assumption with Linux’s 40 million lines of code and 9756 CVEs. Both halves are worth stealing in their account, since a vendor’s fine-tuned weights cost real money to train yet sit on a stranger’s device, and the apps calling a shared model feed it chat history and screen contents. The authors name the boundary they cannot cross, which is that an app in the normal world still receives its own input and output there, so a compromised kernel can take those, and only the weights stay protected in that case. Plain TrustZone is slow at this because it hands out secure memory as a few large contiguous regions. Reserving 8 GB at runtime therefore means the kernel’s contiguous allocator compacting scattered pages, which the authors time at 6440.67 ms against 568.58 ms for FlexServe’s page-granular secure memory, and the secure world cannot normally touch the NPU at all, leaving 30.06 s of CPU computation where the NPU needs 1.94 s. FlexServe carves secure pages out with the stage-2 page tables that ARM’s virtualisation extension already provides, and flips the NPU between protected and unprotected mode in 0.21 ms. Handing 8 GB back drops from 732.85 ms to 80.50 ms, because the pages are released lazily rather than zeroed in one pass. A scheduler decides which models stay resident when several share the device, holding the earliest layers of each in secure memory so the pipeline does not stall and prefetching the next model while the current one is still generating. With a 4 GB cache and requests drawn at random from UltraChat, OpenAssistant, Dolly and Alpaca, the authors measure pairs of models answering 8.84 times faster than the plain secure-world baseline and 2.53 times faster than the optimised one. Across ten agent workflows, where one model’s finished answer feeds the next, the gains reach 24.30 times and 4.05 times, averaging 14.15 and 2.94. Serving one phone-side model to several callers is also what ElastiLM works on, by resizing the shared model to each request’s latency budget rather than isolating it. The protection costs the rest of the device something as well. A SQLite instance holding 3 GB in the normal world runs 1.31 times slower while FlexServe holds 8 GB in the background, against 3.27 times slower with the contiguous allocator, and the authors put the standing virtualisation cost at 2.46 percent on SPEC CPU, which they switch off when no secure inference is running. They list the prototype’s limits themselves, a development board rather than a phone, Llama 3 and Qwen3 series models only, no GPU in the pipeline, and an NPU runtime slow enough that computation becomes the bottleneck, so a faster NPU would widen their lead. Side-channel attacks, physical attacks and denial of service are out of scope, and the paper names no code release. Source: https://arxiv.org/abs/2603.09046 Read the article: https://llmobile.kavents.com/ticker/flexserve/ResearchArmNPUMemoryLlamahttps://llmobile.kavents.com/ticker/clawmobile/https://llmobile.kavents.com/ticker/clawmobile/Thu, 26 Feb 2026 13:34:00 +0100Seven researchers at MBZUAI, City University of Hong Kong and one working independently published ClawMobile on February 26, 2026, an agent runtime that runs on the Android phone itself with no tethering to a host machine. The model is not on the device. The authors state that ClawMobile runs locally while model inference is performed remotely, and all agents in their tests use GPT-5.2, so the split is an on-device runtime driving a cloud model. Across six real-life phone tasks the authors report ClawMobile completing all six at 100 percent, against 33 to 100 percent for the UI agent they compare against. What separates it from screen-reading GUI agents is that the reasoning loop never touches the screen. The orchestrator issues tool calls to control backends instead, and the authors describe a deterministic-first policy in which the runtime first checks its memory for a structured interface, reaching for ADB system commands and the Termux hardware API before anything visual. A semantic UI agent is invoked only when a step genuinely depends on reading dynamic interface state, and raw taps on interface elements sit below that as a fallback. After every action the runtime re-queries device state to confirm the step landed, which the authors contrast with inferring execution success purely through model reasoning. Architecture diagram: Du et al., CC BY 4.0. The tests ran on a Google Pixel 9 on Android 16, with DroidRun as the UI agent baseline and a stripped ClawMobile without DroidRun as a second baseline, all three on GPT-5.2 and scored by human annotators. In the authors’ table DroidRun reaches 100 percent on switching the system to dark theme and on playing a YouTube video, 85 percent on posting a YouTube comment, 73 percent on a Chrome search and on a cross-app task that searches football results and writes them into Notes, and 33 percent on installing an app from the Play Store, where ClawMobile scores 100 percent throughout. The authors put the cost of that at 57.5 seconds slower per task on average, with ClawMobile taking 21 seconds for the dark theme switch and 235 seconds for the YouTube comment. Their deterministic-only variant hits the 600-second timeout on both YouTube tasks. The authors name the remote model as the main open problem, writing that fully remote inference introduces network latency and privacy concerns and calling on-device inference a promising direction they have not taken. They also flag that serialising full UI trees or screenshots into model context can dominate token use and latency on mobile, and present the results as preliminary, six tasks on one device. The code is open source on GitHub, built on the OpenClaw agent framework with a Telegram bot as the chat channel, and the paper was accepted at EuroMLSys 2026 under a Creative Commons Attribution 4.0 license. Source: https://arxiv.org/abs/2602.22942 Read the article: https://llmobile.kavents.com/ticker/clawmobile/AgentsAndroidPixelResearchOpen sourcehttps://llmobile.kavents.com/ticker/nemotron-flash/https://llmobile.kavents.com/ticker/nemotron-flash/Mon, 24 Nov 2025 09:46:00 +0100Nvidia published Nemotron-Flash on November 24, 2025, a family of small language models built around measured decoding latency instead of parameter count. Nvidia reports Nemotron-Flash-1B decoding 8k tokens in 14.45 seconds on a single Nvidia H100 at batch size 1, against 27.55 seconds for Qwen3-0.6B, while averaging 49.63 across 16 benchmarks to Qwen3-0.6B’s 44.11. The paper was accepted at NeurIPS 2025. The models mix three kinds of layer rather than using attention throughout. Nvidia interleaves DeltaNet and Mamba2 layers, operators that carry a fixed-size state instead of a cache that grows with the context, with a small number of ordinary attention layers, and an evolutionary search picked which operator goes where. Nemotron-Flash-1B holds 0.96B parameters in 12 blocks with only 2 full attention layers, and Nemotron-Flash-3B holds 2.7B parameters in 18 blocks with 3. Nvidia built both wider and shallower than other models of the same size, where Qwen3-0.6B stacks 28 layers, and writes that deep and thin models are parameter efficient but sit off the accuracy-latency frontier. Nemotron-Flash-3B averages 60.98 across the same 16 tasks, which cover MMLU, commonsense reasoning, math, coding and recall, against 58.96 for Qwen2.5-3B and 55.47 for Qwen3-1.7B. On the same H100, with all models running through TensorRT-LLM AutoDeploy kernels and CUDA Graph, Nvidia measured 28.71 seconds for the 3B against 49.40 for Qwen2.5-3B and 36.20 for Qwen3-1.7B, so 1.7 and 1.3 times lower latency. Throughput was measured separately, with a 32k-token input at the largest batch size that still fits in memory, and Nvidia reports 2939 tok/s for the 3B against 459 and 157 for the two Qwen models, and 7289 tok/s for the 1B against 160 for Qwen3-0.6B. Every figure here is Nvidia’s own, taken on data centre GPUs, and the paper gives no phone or embedded measurements, with the earlier depth and operator studies run on an Nvidia A100. Nvidia trained both models on 4.5T tokens using 256 H100 GPUs and released three checkpoints on Hugging Face, Nemotron-Flash-1B, the 3B base model and an instruction-tuned Nemotron-Flash-3B-Instruct, which averages 52.84 across MMLU, GPQA, GSM8K and IFEval against 48.17 for Qwen2.5-1.5B. The model cards set the licence to Creative Commons Attribution-NonCommercial 4.0, so the weights are downloadable but not licensed for commercial use, and neither Apache nor an Nvidia open model licence applies. Running them needs the TensorRT-LLM AutoDeploy attention kernel alongside the Mamba2 and FlashLinearAttention implementations, since Nvidia states that vLLM and TensorRT-LLM do not support DeltaNet-style linear attention on their own. Chart: Fu et al., Figure 1 of the paper. Nvidia's own measurements on an Nvidia H100. Source: https://arxiv.org/abs/2511.18890 Read the article: https://llmobile.kavents.com/ticker/nemotron-flash/NvidiaResearchBenchmarksOpen weightshttps://llmobile.kavents.com/ticker/intelligence-per-watt/https://llmobile.kavents.com/ticker/intelligence-per-watt/Tue, 11 Nov 2025 07:33:00 +0100Researchers at Stanford University and Together AI profiled more than 20 local models on 1M real queries and report that at least one model with 20B or fewer active parameters answered 88.7% of them correctly. The best single model in their set, GPT-OSS-120B, reached 71.4% on its own, averaged across the four query collections. Their 2025 efficiency figure comes from that model running on an Apple M4 Max. The paper proposes intelligence per watt, mean task accuracy divided by the mean power the accelerator draws while answering, so a pairing that gets more answers right on the same power budget scores higher. The best local pairing each year moved from 7.92×10⁻⁴ accuracy per watt for Mixtral 8x7B on an NVIDIA Quadro RTX 6000 in 2023, to 1.80×10⁻³ for Llama 3.1 8B on an RTX 6000 Ada in 2024, to 4.18×10⁻³ for GPT-OSS-120B on an Apple M4 Max in 2025, which the authors put at a 5.3 times gain over two years. Query coverage for that single best pairing rose from 23.2% to 71.3% across the same three years. A companion metric divides accuracy by the energy a query consumes instead, so slow generation counts against a device. Diagram: Saad-Falcon et al., Figure 1 of the paper, licensed CC BY 4.0. The hardware panel separates the accelerators the authors treat as local from the cloud parts. The 1M queries come from four collections. The team took 500K prompts from WildChat, a month of real ChatGPT traffic, 500K from NaturalReasoning, 12K from MMLU Pro and 26.5K from SuperGPQA, all single-turn. WildChat has no ground truth, so GPT-4o judged whether a local model’s answer matched one written by Qwen3-235B, while the other three collections supplied reference answers. Every run used a batch size of 1, one query at a time. The hardware the authors call local is desktop and workstation class rather than mobile. Their 2025 numbers ran on an Apple Mac Studio with an M4 Max, 128 GB of unified memory and 480 W of continuous power, and the 2023 and 2024 points used NVIDIA workstation cards. No phone, tablet or wearable was profiled. Running the same Qwen3 models, the M4 Max reached 1.40 times lower accuracy per watt than an NVIDIA B200 and 1.6 to 2.3 times lower accuracy per joule, a gap the authors read as headroom for local accelerator design. For the savings case the team simulated 80.2M queries over 24 hours split between four small models on M4 Max machines and Qwen3-235B in the cloud. Perfect routing to the smallest capable model cut energy by 80.4% against sending everything to the frontier model, and a router that picks correctly 80% of the time cut it by 64.3%, with the perfect case described as a theoretical maximum. The authors note that coverage falls in technical domains such as architecture and engineering, that GPT-OSS-120B answered 88.9% of WildChat queries against 64.9% of NaturalReasoning ones, and that they release their profiling harness so the measurements can be repeated on new hardware. Source: https://arxiv.org/abs/2511.07885 Read the article: https://llmobile.kavents.com/ticker/intelligence-per-watt/ResearchBenchmarksApple SiliconNvidiaQwenhttps://llmobile.kavents.com/ticker/mobilellm-pro/https://llmobile.kavents.com/ticker/mobilellm-pro/Mon, 10 Nov 2025 06:28:00 +0100Meta Reality Labs published the MobileLLM-Pro technical report on November 10, 2025, describing a 1.08B-parameter language model built for phones and wearables. The model handles a context window of 128,000 tokens, four times what Gemma 3 1B and Qwen 3 0.6B accept. Quantising weights and embeddings to 4 bits takes the file from 2.2 GB in bf16 down to 590 MB for the CPU build and 720 MB for the accelerator build, which cannot share embedding weights. Meta exported the model through ExecuTorch and measured it on a Samsung Galaxy S25 CPU with the XNNPACK backend and on a Samsung Galaxy S24 Hexagon Tensor Processor. On a 2,000-token prompt the paper reports 8.9 s of prefill on the CPU against 2.0 s on the Hexagon unit, and decode speeds of 33.6 tok/s and 31.6 tok/s while generating 1,000 tokens. At 8,000 tokens prefill climbs to 63.5 s on the CPU and 9.8 s on the Hexagon unit, decode falls to 19.7 and 22.8 tok/s, and the key-value cache grows from 14 MB to 40 MB. The architecture puts 30 layers of width 1280 behind 20 attention heads and 4 key-value heads, and shares one weight matrix between the input embedding and the output head, which the authors say saves 260M parameters, about a quarter of the model. Three out of every four layers use local attention, comparing each token only against the 512 most recent ones, with a global layer every fourth one and at both ends. Meta’s model card credits that interleaving with cutting the key-value cache at 8,000 tokens from 117 MB to 40 MB and with 1.8 times lower prefill latency than global attention in every layer. Meta trained the model by distilling the logits of Llama 4 Scout under a KL divergence loss across three pretraining phases on roughly 1.64T tokens of open data, the second of them a step the authors call implicit positional distillation, which stretches the context window to 128k by having the teacher pass on the ability rather than feeding the student long documents. Meta puts the base model 5.7 and 7.9 points above Gemma 3 1B and Llama 3.2 1B on average across the benchmarks its model card lists, and the paper’s table has it at 76.24 on BoolQ against 63.20 and 62.51, at 52.62 on ARC-Challenge against 38.40 and 38.28, and at 100% needle-in-a-haystack retrieval against 96.80 for Llama 3.2 1B. The instruction-tuned version scores 59.8 on HumanEval against 41.5 and 37.8, 44.8 on MMLU against Gemma’s 29.9 and Llama’s 49.3, and 62.0 on IFEval against Gemma’s 80.2. In Meta’s own human evaluation over 100 prompts per task, raters preferred Gemma 3 1B for summarisation 51 to 47 and for rewriting 49 to 45, while MobileLLM-Pro led on recall 47 to 30. Meta released four checkpoints on Hugging Face, the base model, the instruction-tuned one, and two quantisation-ready base checkpoints produced by quantisation-aware training with self-distillation from the full-precision model. One uses 4-bit group-wise weights at group size 32 with 8-bit dynamic activations and an 8-bit key-value cache for CPU backends, the other 4-bit channel-wise weights for the Apple Neural Engine and Qualcomm’s Hexagon processor, which handle group-wise quantisation poorly or not at all. Needle-in-a-haystack retrieval holds at 96.44% and 98.67% against 100% at full precision. The weights sit behind a request form and carry the FAIR Noncommercial Research License. Source: https://arxiv.org/abs/2511.06719 Read the article: https://llmobile.kavents.com/ticker/mobilellm-pro/MetaResearchDistillationQuantisationAndroidhttps://llmobile.kavents.com/ticker/confidant-collaborative-training/https://llmobile.kavents.com/ticker/confidant-collaborative-training/Mon, 03 Nov 2025 10:00:00 +0100Researchers at Zhejiang University, Zhejiang University of Technology and Hangzhou Dianzi University published Confidant in the ACM MobiCom 2025 proceedings on November 3, 2025, a day before the conference opened in Hong Kong. The framework fine-tunes a language model across several off-the-shelf devices instead of one, and the authors fine-tuned Phi2-2.7B on the Alpaca instruction dataset in 40.1 hours using three of them. Their write-up of that run names a Redmi K50 phone, a Core i7-13650HX laptop and a MacBook Pro with an M1 Pro, while the paper’s abstract and conclusion describe the same result as three mobile devices. The same trio fine-tuned BERT-Base in 2.2 hours. Confidant cuts a model into consecutive slices, gives each device one slice, and passes activations from one device to the next, so no device ever holds the whole model. Batches flow through that chain as a pipeline, with each device starting a new batch while the devices behind it are still working on earlier ones. The device that holds the training data acts as coordinator and recalculates where the cuts fall during training, using each participant’s measured compute time, free memory and bandwidth, and it discards any split whose slice would not fit in a device’s memory. Inside a device, a second scheduler profiles the CPU and the GPU on varying numbers of attention heads and gives each processor as many heads as it can finish in roughly the same time, which is what the authors credit for running up to 1.94 times faster than a capacity-aware pipeline baseline they implemented themselves, and up to 3.82 times faster than the best single device, both on GPT-2 Medium. Diagram: Chen et al., Figure 3 of the paper, licensed CC BY 4.0. Splitting the model is what brings the footprint per device down. The paper’s memory table puts fine-tuning BERT-Base at a batch size of 8 at 5.5 GB on a single Redmi K50, against a per-device average of 3.17 GB across three devices and 2.43 GB across four. Larger models leave no single-device figure to compare with, because a Redmi K50 fits only 1 of Phi2-2.7B’s 32 transformer blocks at a batch size of 4, and the per-device averages of 15.31 GB for Phi2-2.7B and 32.07 GB for LLaMA3-8B across three devices are carried by the two laptops rather than the phone. A separate table puts the memory PyTorch needs to fine-tune Phi2-2.7B at a batch size of 8 at 46.5 GB, next to the 8 GB to 12 GB of the four phones it lists. The paper’s prose says Confidant almost does not cause any accuracy drop, while its own model performance table puts the three-device runs at a test loss of 0.66 on Phi2-2.7B and 3.55 on LLaMA3-8B, against 0.52 and 2.86 for fine-tuning the same models the conventional way on a single machine, where a lower loss is better. The four-device runs land at 0.76 and 2.85 on that same comparison, and BERT-Base token classification accuracy holds at 97.66% against 97.70%. Keeping data on the device is the stated motivation. The authors write that training with Confidant does not require sharing raw data and that devices exchange only intermediate outputs and model gradients, which they present as the privacy argument for their two scenarios, a household assistant fine-tuned on family data during idle hours and a conference assistant fine-tuned on attendees’ phones. They also state that fine-tuning on a single cloud server would be faster and more straightforward, and that their aim is to use idle mobile hardware instead. The Android side is built on Alibaba’s MNN framework with laptops running PyTorch, about 10,900 lines of code in total, the paper is published under a Creative Commons Attribution 4.0 license, and the authors wrote that they planned to release the code in late 2025. Source: https://dl.acm.org/doi/10.1145/3680207.3723487 Read the article: https://llmobile.kavents.com/ticker/confidant-collaborative-training/ResearchMemoryPhiLlamaAndroidhttps://llmobile.kavents.com/ticker/lm-meter/https://llmobile.kavents.com/ticker/lm-meter/Tue, 07 Oct 2025 19:05:00 +0200Researchers at Georgia State University and Toyota InfoTech Labs published lm-Meter, a latency profiler that runs inside the inference engine on the phone and splits each generation into embedding, prefill, decode, softmax and sampling. Measuring the Pythia models on a Google Pixel 8 Pro, they report that scaling from 70M to 1.4B parameters raises prefill latency from 0.012 s to 1.9 s per input token, a 158x slowdown, while decode latency per output token grows from 0.015 s to 0.15 s, a 10x slowdown. The authors write that this inverts the server picture, where decode is usually the limiting phase for single-request inference. Below the phase level the profiler times individual GPU kernels through OpenCL event timestamps, which give queue, submit, start and end times without access to the closed-source driver. Running a 4-bit quantised Gemma-2-2B-it on a Pixel 8 Pro, the authors report that fused matrix-multiplication kernels dominate a decode step and that the GPU sits idle for more than 21% of it, the second-largest contributor to the step, which they attribute to host-side data preparation and I/O stalls. The paged attention kernel that scans the growing key-value cache is the only one whose cost rises with position in the sequence, climbing from roughly 0.2 ms to about 0.8 ms per token over 250 decode steps, and idle time drops from about 21% to 12% when the model generates 256 tokens instead of 16. Whether those measurements mean anything depends on what the profiler itself costs. lm-Meter sits in the MLC LLM runtime and TVM in about 3,500 lines of code and needs no host machine attached, and under the Powersave CPU governor, the most constrained setting they tested, the authors measure a throughput loss of 2.58% in prefill and 0.99% in decode. They put the same figures for MELTing Point, the on-device profiler they compare against, at 22% for prefill and more than 93% for decode. Checked against traces from Android GPU Inspector, they report end-to-end phase accuracy of at least 99.99% and mean kernel-level accuracy of 96.82% on the Pixel 8 Pro and 96.61% on a Pixel 7. The code is on GitHub under the MIT license, with the MLC LLM path released for Android GPUs through OpenCL and support for llama.cpp, vLLM, iOS Metal and Nvidia Jetson listed as unfinished. The work was accepted to the ACM/IEEE Symposium on Edge Computing 2025 and funded by Toyota Motor North America. The measurements come from three phones, the Pixel 8 Pro, Pixel 7 and Pixel 6, and the authors state that other edge platforms such as Jetson boards and Intel NPUs may show different bottlenecks. Source: https://arxiv.org/abs/2510.06126 Read the article: https://llmobile.kavents.com/ticker/lm-meter/ResearchBenchmarksDeveloper toolsMLC LLMPixelhttps://llmobile.kavents.com/ticker/mobilellm-r1/https://llmobile.kavents.com/ticker/mobilellm-r1/Fri, 12 Sep 2025 12:00:00 +0200Meta released MobileLLM-R1 on September 12, 2025, a family of reasoning models at 140M, 360M and 950M parameters that write out a chain of intermediate steps before giving an answer instead of replying directly. Meta reports the 950M model scoring 74.0 on the MATH500 maths benchmark and 19.9 on LiveCodeBench v6, against 73.0 and 14.9 for Qwen3-0.6B. The technical report followed on September 29, 2025. The three models saw 4.2T training tokens, which Meta puts at 11.7% of the 36T tokens behind Qwen3-0.6B. Pre-training ran in two phases of 2T tokens each, resampled from a curated pool of roughly 2T tokens of open datasets, followed by two mid-training phases of 100B tokens that distil from Llama-3.1-8B-Instruct as a teacher, then supervised fine-tuning on reasoning traces. Meta uses the result to argue that reasoning does not require the corpora above 10T tokens that recent models have been trained on. For the post-trained models Meta’s table puts MobileLLM-R1-950M at 74.0 on MATH500, 67.5 on GSM8K, 15.5 on AIME 2024 and 19.9 on LiveCodeBench v6, where Qwen3-0.6B reaches 73.0, 79.2, 11.3 and 14.9 and DeepSeek-R1-Distill-Qwen-1.5B reaches 83.2, 77.3, 29.1 and 19.9. Among the models whose weights, data and recipe are all public, the same AIME figure compares with 0.6 for OLMo-2 1.48B and 0.3 for SmolLM2-1.7B. On code the 950M base model scores 46.3 on HumanEval against 30.5 for Qwen3-0.6B-Base, and the 140M base model reaches 15.9 where SmolLM2-135M scores 0.0. Meta profiled the models through ExecuTorch on a Samsung Galaxy S22 with 8 GB of RAM, quantised to 4-bit weights with 8-bit dynamic activations. The 140M model generated 129.67 tok/s at a 1k context and 79.71 tok/s at 32k, the 950M model 31.05 tok/s at 1k before running out of memory at 16k, and Llama-3.2-1B 28.71 tok/s at 1k with the same memory failure. The architecture follows Meta’s earlier MobileLLM work and Llama 3.2, sharing one weight matrix between the input embedding and the output layer, and the post-trained models take a 32k context. Meta released the weights on Hugging Face together with the training code, the data sources and the data mixing ratios on GitHub, under the FAIR Noncommercial Research License, which covers research, education and analysis but not commercial use. Access to the model repositories is gated behind a form. Update, November 24, 2025. Meta released MobileLLM-R1.5 at the same three sizes, fine-tuned from the R1 models with one further round of on-policy distillation, where the student answers first and the teacher, here Llama-3.1-Nemotron-Nano-4B-v1.1, grades its own words back to it token by token. Meta reports the 950M version at 86.6 on MATH500, 82.6 on GSM8K, 39.9 on AIME 2024 and 29.1 on LiveCodeBench v6, up from 74.0, 67.5, 15.5 and 19.9 for MobileLLM-R1-950M, and says it beats DeepSeek-R1-Distill-Qwen-1.5B on every maths and coding benchmark it ran. At 360M Meta puts MATH500 at 63.4 against 28.4 and GSM8K at 52.8 against 24.5. Source: https://arxiv.org/abs/2509.24945 Read the article: https://llmobile.kavents.com/ticker/mobilellm-r1/MetaResearchBenchmarksDistillationAndroidhttps://llmobile.kavents.com/ticker/shadownpu/https://llmobile.kavents.com/ticker/shadownpu/Fri, 22 Aug 2025 09:41:00 +0200Researchers from Peking University and BUPT report up to 4.5 times faster end-to-end inference on a Xiaomi 14 with a Snapdragon 8 Gen 3, and 3.5 times faster on average for the attention kernel alone, with a peak of 6.9 times. The baseline is the arrangement most on-device frameworks use today, float32 full attention running on the CPU or GPU while the rest of the model sits on the NPU. The authors measured Qwen2-0.5B, Qwen2-1.5B, PhoneLM-0.5B and PhoneLM-1.5B on summarisation and two mobile agent tasks, and gave their own system only one of the phone’s middle CPU cores. Against the native attention of llm.npu, which uses four CPU cores or the GPU, they report up to 3.0 times lower latency from that one core. The paper is due at MobiSys 2026. The starting problem is that attention keeps falling back off the NPU. Attention multiplies three activation tensors together, and mobile NPUs compile a static graph that fixes one quantisation scale factor for a whole tensor, so the authors measure an average 18 percentage point accuracy drop when the operator runs in INT8 on the NPU. Their trade is to stop asking the NPU for the answer and ask it only for the ranking. The NPU computes the query-key scores in INT8 purely to find which tokens matter, which the authors report recovers over 99 percent of the positions a float run would pick, since picking the largest values needs relative magnitudes rather than exact ones. The CPU then computes real attention in float over roughly 20 percent of the tokens and throws the other 80 percent away, at an average accuracy cost of 0.4 percentage points against the float baseline. Each attention head gets its own sparsity ratio, fixed by an offline profiling run the paper puts at about five minutes on a single A100. On a Redmi K60 Champion Edition with a Snapdragon 8 Gen 2 the authors read battery voltage and current from the kernel and report a single attention kernel at 1024 tokens dropping from 8.59 J to 1.12 J for PhoneLM-1.5B, a factor of 7.66. Speedups on that older phone are smaller, 2 times, 1.25 times and 1.22 times across the three datasets, which the authors attribute to its Hexagon V73 NPU against the V75 in the Xiaomi 14. The paper reports no memory footprint figures. It does report that end-to-end inference time on the Xiaomi 14 stayed between 14.5 and 15.5 seconds while the phone was also taking photos, playing video, playing music or running a shopping app, since the LLM work is not competing for the cores those apps use. Diagram: Yin et al. The authors state a prototype of more than 10,000 lines of C++ and Python, built on Qualcomm’s QNN and Hexagon SDK for the NPU side and integrated into llm.npu, which they call the only fully open-source framework running on NPUs. The paper names no repository and announces no artefact release, and the title calls the system ShadowNPU while the text throughout calls the module shadowAttn. The arXiv listing carries a CC BY 4.0 license. For a hands-on point of comparison on the same silicon family, a user test of llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone put generation at about the same speed as the chip’s CPU, without the heat. Source: https://arxiv.org/abs/2508.16703 Read the article: https://llmobile.kavents.com/ticker/shadownpu/QualcommNPUResearchQwenAndroidhttps://llmobile.kavents.com/ticker/pd-device-cloud-split/https://llmobile.kavents.com/ticker/pd-device-cloud-split/Tue, 12 Aug 2025 17:56:29 +0200Thirty researchers at Huawei Technologies published P/D-Device on August 12, 2025, a scheme that runs a single language model request across a cloud data centre and a phone at the same time instead of choosing one of them. The cloud reads the prompt and writes only the first few tokens, then the device carries the answer to the end. The authors measure average time to first token falling by at least 60 percent against running the whole request on the device, and cloud throughput rising by 1.6 to 15 times, 7.6 times on average, against letting the cloud generate a full 200-token answer. The split follows from the two phases of inference wanting different hardware. Prefill, where the model reads the prompt before it writes anything, is compute-bound, and the authors report on-device time to first token climbing steeply with prompt length, into seconds or tens of seconds once prompts run to several thousand tokens. Decoding, where the model emits one token at a time, behaves differently, because a phone serves one user while a cloud instance serves a large batch, so the authors find the two roughly matched at tens of milliseconds per output token. Decoding is also what occupies a cloud instance for tens of seconds per request, so the scheme cuts the cloud off after a set number of tokens and hands the rest of the answer to the device. The device displays the cloud’s first token the moment it arrives, so the wait the user actually perceives is the cloud’s prefill, which the authors put at hundreds of milliseconds. Behind that display the device runs its own prefill while a speed controller releases the next tokens from the cloud at a deliberately slowed pace, matched to human reading speed, until the device has caught up and generates the rest itself. During its prefill the cloud also scores the prompt using the attention weights it has already computed, picks the sentences that matter most, and returns the selection as a compressed mask so the device prefills a shorter prompt, with a quarter of the prompt kept by default. What crosses the network is text, never model state. The device sends the prompt as raw text and the cloud returns the first token, the mask and a token budget, which the authors cap at hundreds of kilobytes for an 8k prompt, while the KVCache, the gigabyte-scale intermediate state that other cloud-device schemes ship back and forth, is too large to move in time and is never transferred at all. On the prompt text itself the authors state that the request goes out only under user and app authorisation and after sensitive information such as phone numbers and real names has been removed or substituted, and they cite Apple’s Private Cloud Compute as the class of guarantee they have in mind, without describing an implementation or measuring one. The archive’s post on Flower Intelligence covers the reverse arrangement, where the device runs first and the remote service stays off unless the app turns it on. Diagram: Jin et al. The cloud scores the prompt during its own prefill and returns the selection as a mask alongside the first token. The prototype runs cloud inference on Huawei Ascend NPUs and device inference on Kirin SoCs, tested on a phone and a tablet, with every chip and model anonymised as Type 1, Type 2, Model 1 and Model 2 throughout the paper. On LongBench with the same model on both sides, the authors’ own table gives their refinement an average of 26.64 on single-document QA against 25.89 for the unrefined on-device baseline, and 26.83 against 22.96 on multi-document QA. The authors name the costs themselves, a quality score around 85 percent of cloud-only inference when the cloud runs the larger model, refinement adding one or two hundred milliseconds at an 8k prompt and several hundred under load, and the smoothed display running slightly above their own tolerable threshold of about a hundred milliseconds per token once prompts pass 8k. The paper, on arXiv under a CC BY 4.0 license, further states that connections were tested over Wi-Fi with an IP whitelist rather than LTE, that collaboration across different NPU types is still being explored, and that the device supports prompts of several thousand tokens against hundreds of thousands in the cloud. Source: https://arxiv.org/abs/2508.09035 Read the article: https://llmobile.kavents.com/ticker/pd-device-cloud-split/ResearchHuaweiNPUBenchmarkshttps://llmobile.kavents.com/ticker/demystifying-slms-edge/https://llmobile.kavents.com/ticker/demystifying-slms-edge/Sun, 27 Jul 2025 09:00:00 +0200Ten researchers at Beijing University of Posts and Telecommunications, Peng Cheng Laboratory, the University of Cambridge, Xiaomi, Helixon Research and Flower Labs collected 68 small language models between 100M and 5B parameters, released by 24 organisations between OPT in May 2022 and Llama 3.2 in September 2024, and measured both their accuracy and their cost on edge hardware. In Demystifying Small Language Models for Edge Deployment, presented at ACL 2025, Microsoft’s Phi-3 and Phi-3.5-mini sit highest on the capability chart at about 70 percent average accuracy over 10 reasoning and knowledge datasets, level with the Llama 3.1 line the authors draw for comparison. The runtime half of the study covers the 20 models that llama.cpp supports, quantised to 4-bit and run with a 50-token prompt and 50 generated tokens. Chart: Lu et al., licensed CC BY 4.0. Accuracy is averaged over the 10 datasets with math excluded, and circle size is proportional to model size. Parameter count predicts on-device latency poorly. The measurements put OpenELM-3B at 9.4 percent more parameters than Phi-2 but 9.9 percent faster to the first token, and Qwen1.5-0.5B at 25.4 percent more parameters than Qwen2-0.5B yet 31.9 percent faster to the first token and 11.8 percent faster per decoded token. Qwen1.5-1.8B carries 30.5 percent more parameters than Pythia-1.4B and still reaches the first token 18.5 percent sooner. The authors trace the gap to how a model spends its parameter budget, with Qwen2 sharing its input embedding and output layer and putting the saved weights into wider attention and feed-forward layers, which costs compute in the prefill stage. Memory ran from 275 MB to 2456 MB across the 20 models, measured with llama.cpp on a Jetson Orin NX at a fixed 2048-token context. Vocabulary size moves that figure as much as parameter count does, and the paper reports Bloom-560M needing 492 MB of compute buffer for its 250,880-token vocabulary, 3.5 times what OpenELM-1.1B needs with 32,000 tokens. OpenELM-1.1B uses 23.8 percent less memory than Bloom-560M despite 32.3 percent more parameters, and OpenELM-3B 13.8 percent less than Gemma-2B despite 21.1 percent more, which the authors put down to the small vocabulary and to grouped-query attention, where several attention heads share one set of cached keys and values. Raising each model to its own maximum context window pushes Qwen2-0.5B past 6 GB, with compute buffer and KV cache making up 85 percent of the total for the Qwen2 models. Charts: Lu et al., licensed CC BY 4.0. Measured with llama.cpp at 4-bit, and memory taken on a Jetson Orin NX at a 2048-token context. Quantisation paid off in decoding rather than in prefill. Testing five settings on Phi-1.5, the authors found Q4_K_M the strongest, roughly halving per-token decode latency against FP16, while at a 50-token prompt it cut the time to the first token by only 13 percent. Q6_K and Q3_K matched or exceeded FP16 on longer prompts, which the paper puts down to the alignment and padding their irregular bit widths force. The hardware gap was wider than the quantisation gap, with the Jetson’s GPU 40 times faster than the Meizu 18 Pro’s CPU at prefill but only 1.84 times faster at decode, and the Xiaomi 12S on a Snapdragon 8 Gen 1+ the fastest of the three phones, ahead of the Pixel 7 Pro and the Meizu on its Snapdragon 888. The authors name two limits on their own work. They left math datasets out of the capability evaluation because of the gap between small models and larger ones on mathematical reasoning, and they confined the cost analysis to the 20 models llama.cpp supports so that the inference engine stayed constant, excluding every model it cannot load. The paper is free to read at the ACL Anthology under a Creative Commons Attribution 4.0 licence, and the same group had earlier measured 22 models from 0.5B to 7B on four Android phones, three of which reappear in this test bed. PalmBench adds power draw and surface temperature on comparable hardware, and Artificial Analysis has since reported quality, speed and memory together for 33 quantised models on one iPhone 17 Pro. Source: https://aclanthology.org/2025.acl-long.718/ Read the article: https://llmobile.kavents.com/ticker/demystifying-slms-edge/BenchmarksResearchllama.cppMemoryAndroidhttps://llmobile.kavents.com/ticker/generative-ai-at-the-edge-queue/https://llmobile.kavents.com/ticker/generative-ai-at-the-edge-queue/Wed, 21 May 2025 18:00:00 +0200Vijay Janapa Reddi of Harvard University published an article in ACM Queue on May 21, 2025, arguing that generative models reach phones, glasses and robots only if the field stops chasing parameter counts and starts designing for deployment. He projects more than 50 billion edge devices by 2030 and writes that today’s frontier models, often hundreds of billions of parameters, cannot leave the data centre. His deployment matrix places only models under 1B parameters in the viable column for true edge hardware, puts medium models of 1B to 10B parameters within reach of high-end phones and single GPUs, and confines anything larger to servers or the cloud. Reddi grounds that in memory first. He writes that a 6B parameter model at half precision needs roughly 12 GB for weights alone, far above typical mobile RAM, that even a 1B model at about 2 GB can strain some devices, and that small models usually occupy 100 MB to 2 GB. He puts the latency bar at around 50 ms for an augmented reality translation to feel instantaneous and under a second for a dialogue assistant to feel fluid, and he notes that batching across users, which keeps cloud serving efficient, does not apply when requests arrive one at a time. On energy he cites an int8 quantised model using about five times less energy per query than an FP16 one on the same hardware, and says MLPerf Power results suggest model energy efficiency is plateauing even as hardware improves. Reddi frames the whole problem as a data-model-compute triangle, where scarce local data, limited compute and the need for compact models each worsen the others, and he argues compression alone will not close the gap, since smaller models hallucinate more and can lose their refusal behaviour when quantised hard. He proposes measuring capability and cost together through metrics such as hallucinations per watt-hour, running a model on factual question answering until it consumes 1 Wh and counting the wrong statements, and toxicity per token normalised by model size. The article ran in Queue volume 23, issue 2, and the full text is free to read on queue.acm.org. Source: https://dl.acm.org/doi/abs/10.1145/3733702 Read the article: https://llmobile.kavents.com/ticker/generative-ai-at-the-edge-queue/ResearchMemoryQuantisationNPUWearableshttps://llmobile.kavents.com/ticker/qwen-3/https://llmobile.kavents.com/ticker/qwen-3/Wed, 14 May 2025 15:41:00 +0200Alibaba’s Qwen Team published the Qwen3 Technical Report on May 14, 2025, covering six dense models from 0.6B to 32B parameters and two mixture-of-experts models, where only part of the model is active per token. The two smallest dense sizes, Qwen3-0.6B and Qwen3-1.7B, carry the same mode switch as the rest of the family, so one set of weights either works through a problem step by step before answering or replies straight away. Alibaba reports Qwen3-1.7B scoring 93.4 on MATH-500, a 500-problem maths set, with reasoning on, against 73.0 with it off. Earlier Qwen releases needed two separate models for that, a chat model and a dedicated reasoning model such as QwQ-32B. In Qwen3 the user picks the behaviour with a flag in the prompt or the system message, /think or /no_think, and the report states that reasoning is the default and that across a multi-turn conversation the model follows the last flag it saw. The same section describes a reasoning budget, where the model’s working is cut off once it passes a token limit the user set and it answers from what it has so far, which the authors say they never trained for and which emerged from merging the two modes. The benchmark tables give both small sizes in both modes. For Qwen3-0.6B, Alibaba reports 55.6 on MMLU-Redux, a corrected version of the MMLU knowledge test, with reasoning on against 44.6 with it off, and 77.6 against 55.2 on MATH-500. For Qwen3-1.7B the same pair of tables gives 73.9 and 64.4 on MMLU-Redux, 48.3 against 13.4 on the AIME 2024 competition maths set, and 33.2 against 11.6 on LiveCodeBench v5. Against the reasoning baselines in the report, Qwen3-0.6B is ahead of DeepSeek-R1-Distill-Qwen-1.5B on MMLU-Redux, 55.6 to 45.4, and behind it on MATH-500, 77.6 to 83.9. Alibaba built the small sizes by copying behaviour from the large ones rather than by running the full reinforcement learning pipeline on each. The report describes two stages of distillation from Qwen3-32B and Qwen3-235B-A22B, first on stored teacher outputs generated in both modes, then by matching the small model’s output probabilities against the teacher’s while the small model generates, and it credits the first stage with teaching the smaller models to switch modes at all. Pre-training ran on about 36 trillion tokens covering 119 languages and dialects, up from 29 in Qwen2.5. Context length is where the two small sizes sit apart from the family. Table 1 of the report puts Qwen3-0.6B and Qwen3-1.7B at 32K tokens against 128K for Qwen3-8B and every larger model, and both tie the input embedding to the output layer so one weight matrix does both jobs. All eight models are published under the Apache 2.0 license on Hugging Face, ModelScope and Kaggle. The Qwen3 blog post of April 29, 2025 names Ollama, LM Studio, MLX, llama.cpp and KTransformers as the tools it recommends for running them locally. Source: https://arxiv.org/abs/2505.09388 Read the article: https://llmobile.kavents.com/ticker/qwen-3/AlibabaQwenOpen weightsResearchBenchmarkshttps://llmobile.kavents.com/ticker/scaling-on-device-gpu-inference/https://llmobile.kavents.com/ticker/scaling-on-device-gpu-inference/Thu, 01 May 2025 02:44:00 +0200Eight researchers at Google published ML Drift on May 1, 2025, a GPU inference framework for running large generative models on the graphics chip of a phone, a laptop or a Mac. On the Adreno 750 GPU of a Samsung S24, the authors measure 1370 tokens per second of prefill and 37.1 tokens per second of decode for Gemma2 2B, and 412 and 12.7 tokens per second for Llama 3.1 8B. They report a 5 times to 11 times prefill speedup over llama.cpp, MLC LLM and the Qualcomm AI Hub benchmark on Qualcomm GPUs, and the work is due at the CVPR 2025 Workshop on Efficient and On-Device Generation (EDGE). The team benchmarked five mobile GPUs, the Adreno 830 in a Xiaomi 15 Pro with 16 GB of RAM, the Adreno 750 in a Samsung S24 with 8 GB, the Adreno 740 in a Samsung S23 Ultra with 8 GB, the Arm Immortalis-G720 in a vivo X100 Pro with 16 GB and the Arm Mali-G715 in a Pixel 9 with 12 GB, each at a fixed 1024 prefill and 256 generated tokens. Llama 3.1 8B with every weight at 8-bit ran out of memory on the S24, the S23 Ultra and the Pixel 9 and finished only on the two 16 GB phones, at 7.70 and 4.72 tokens per second of decode. A mixed scheme that holds attention weights at 8-bit and drops embedding and feed-forward weights to 4-bit ran the 8B model on all five devices and lifted decode by up to 1.9 times. ML Drift also treats the two phases of generation separately, since filling the prompt is limited by arithmetic while producing each further token is limited by memory bandwidth. Chart: Google. Solid bars are prefill on the left axis, cross-hatched bars decode on the right axis. Gaps mark runs the authors could not measure. A mobile GPU constrains inference in ways the paper works through one at a time. Memory comes first, and the authors report that Stable Diffusion 1.4 would need 4.31 GB for its intermediate 16-bit activations, which handing the same buffers to operations whose lifetimes do not overlap cuts to 387 MB. Weight layout comes second, and rearranging weight tensors into slices of four channels that match the 4-element units a GPU computes with buys up to a 20% speedup in matrix multiplication, which the authors call the primary driver of the framework’s advantage. Shader code comes third, and ML Drift writes its kernels once at startup from hand-tuned templates, so the translation between a tensor’s logical indices and the buffer or texture that actually holds it costs nothing while tokens are being produced. The same engine renders Stable Diffusion 1.4 at 512 by 512 pixels over 20 iterations in 10.96 seconds on the S23 Ultra, under 9 seconds on the S24 and in 3.4 seconds on a laptop with an Intel Lunar Lake Ultra 7 258V. On an NVIDIA RTX 4090 the OpenCL path runs 5% to 25% slower than llama.cpp on CUDA, which the authors put down to NVIDIA’s Tensor Cores being unreachable through OpenCL and WebGPU. The paper names sub-channel quantisation, sparsity and vendor extensions for matrix multiplication as the next steps, and points to no public release of the framework. Source: https://arxiv.org/abs/2505.00232 Read the article: https://llmobile.kavents.com/ticker/scaling-on-device-gpu-inference/GoogleQualcommAndroidBenchmarksResearchhttps://llmobile.kavents.com/ticker/d2moe/https://llmobile.kavents.com/ticker/d2moe/Thu, 17 Apr 2025 07:37:35 +0200Researchers at the Hong Kong University of Science and Technology and Shenzhen University published D2MoE on April 17, 2025, a serving system for mixture-of-experts models on edge hardware, accepted at ACM MobiCom 2025. On a laptop with a 6 GB NVIDIA RTX 3060 and on a Jetson AGX Orin with 64 GB, the authors measure Mixtral 8x7B at 1.14 to 1.39 times the throughput of EdgeMoE and LLaMA-MoE-3.5B at 1.06 to 1.16 times, the second one while using 33 to 53 percent less memory. Against MoQE-DynaIO, which quantises every expert to one fixed bit-width and loads them on demand, they report 1.42 to 3.37 times the throughput. A mixture of experts replaces the feedforward block in each layer with a set of small networks, and a router sends every token to only a few of them, so the work per token stays small while every expert still has to be held in memory or fetched from somewhere. The paper puts Mixtral 8x7B at over 90 GB in float16, with experts accounting for 89.9 percent of the parameters, against the 6 GB the RTX 3060 laptop offers. One answer is to keep the whole expert set small enough to stay resident, the way Ai2 sizes OLMoE and Meta AI sizes MobileMoE. D2MoE takes the other one, quantising experts hard and loading them from lower tiers on demand, which is also how Edge0 fits a large model into a small budget. That trade turns a memory problem into a transfer problem. On the RTX 3060 the authors time a single LLaMA-MoE expert at 3.1 ms of computation against roughly 20 ms of data transfer, so the GPU sits idle waiting for weights, and holding several quantised copies of each expert to choose from would give the memory back. D2MoE therefore routes each token twice, once to an expert and once to a bit-width for that expert, and nests the quantised weights so that a wider one contains the narrower ones, which the authors name after matryoshka dolls. They note that 4-bit LLaMA-MoE experts quantised with llama.cpp take 3.81 GB, while storing separate 2-bit, 3-bit and 4-bit copies takes 9.62 GB. A scheduler then orders expert loads hottest first so that transfers overlap computation inside a set memory budget, and the paper puts the added cost of the second router at 0.28 percent of computation, 0.53 percent of memory and 1.67 percent of latency for LLaMA-MoE-3.5B. On WikiText2 the configuration meant to match 8-bit experts scores 4.09 perplexity on Mixtral 8x7B, against 4.04 for the baseline that holds all 8-bit experts in memory and 4.38 for EdgeMoE. Setting a model up is offline work, at about 2 hours of router fine-tuning plus 10 minutes of quantisation for LLaMA-MoE-3.5B and over 4 hours plus 20 minutes for Mixtral 8x7B. The prototype is roughly 2,500 lines of Python and CUDA on top of PyTorch, written for NVIDIA Ampere and Ada Lovelace, and the paper names no code release and reports no measurements on phones. Chart: Wang et al. The authors' own measurements on the RTX 3060 laptop, where RN is the number of concurrent requests. Each baseline appears as a single point at the budget it needs, while D2MoE is plotted across budgets. Source: https://arxiv.org/abs/2504.15299 Read the article: https://llmobile.kavents.com/ticker/d2moe/Mixture of expertsMemoryQuantisationNvidiaResearchhttps://llmobile.kavents.com/ticker/roma-qlora-accelerator/https://llmobile.kavents.com/ticker/roma-qlora-accelerator/Mon, 17 Mar 2025 10:44:00 +0100Researchers at Shanghai Jiao Tong University, Peking University and Microsoft Research Asia described ROMA on March 17, 2025, an accelerator design that holds a quantised language model in on-chip read-only memory. The paper reports results from Verilog synthesis against TSMC’s 7 nm standard cell library, not from a fabricated chip and not from an FPGA prototype, so every speed, area and power figure in it is a toolchain estimate rather than a measurement on hardware. On that basis the authors report peak decoding of 31,800 tok/s for a 4-bit 3B Llama model with an empty KV cache. The design splits storage by how often the weights change. The frozen base model, quantised to 2-bit or 4-bit, sits in ROM, which cannot be rewritten once the chip has been manufactured, while the small trainable LoRA adapter and the KV cache sit in SRAM, which can. The authors argue that a base model is stable enough for that, with updates expected over years rather than months, and that adaptation to new tasks happens in the adapter instead. What ROM buys is density, and the paper puts a ROM bit cell at roughly a third of the area of an SRAM bit cell in TSMC’s 7 nm memory compiler, enough to fit the whole quantised model on chip without external memory. The synthesised configuration carries 1.86 GB of ROM and 304 MB of SRAM at 500 MHz, with an area of 503.7 mm² and power of 33.1 W, the area and latency coming from Synopsys Design Compiler and the power from Synopsys PrimeTime PX. It holds either a 4-bit 3B or a 2-bit 8B Llama model entirely on chip, and for the 3B model with a rank-16 adapter the authors report a time to first token of 5.6 ms at 256 input tokens and 140.2 ms at 4K, with decoding staying above 10,000 tok/s once 4K tokens are cached. They also compare against measured hardware, an Intel i5-1135G7 running llama.cpp at 6.8 tok/s and an Nvidia RTX 4090 running TensorRT-LLM at 219 tok/s on the same 4-bit 3B model, against 20,078 tok/s for the ROMA estimate at a 256-token input. Most of the paper’s engineering goes into shrinking the ROM. A structure the authors call B-ROM stores the constant bits in the chip’s wiring instead of in transistors, which cuts the transistor count to about a quarter and the ROM area by around 40 percent against a standard ROM, and a fused cell layout places that wiring-heavy structure in the same physical area as the transistor-heavy compute logic. The paper states no tapeout, no fabrication schedule and no cost figures, and it names the fixed base model as the price of the approach, since ROM content is set at manufacture. It is posted under the arXiv perpetual non-exclusive license. Source: https://arxiv.org/abs/2503.12988 Read the article: https://llmobile.kavents.com/ticker/roma-qlora-accelerator/ChipsMemoryQuantisationResearchLlamahttps://llmobile.kavents.com/ticker/on-device-inference-tradeoffs/https://llmobile.kavents.com/ticker/on-device-inference-tradeoffs/Wed, 12 Mar 2025 08:01:00 +0100Maximilian Abstreiter, Sasu Tarkoma and Roberto Morabito of the University of Helsinki and EURECOM measured 11 language models from 0.5B to 9.2B parameters on a Raspberry Pi 5 and an NVIDIA Jetson Orin Nano Developer Kit, both with 8 GB of RAM, running every model through llama.cpp at 4-bit. On the Pi, the authors report that only models up to Llama 3.2 3B, or up to Phi 3.5 mini in the faster of the two 4-bit formats they tested, generated text above 5.3 tokens per second, the rate they derive from average English reading speed. The Orin’s GPU stayed above that rate for all 11 models, its CPU cores up to InternLM 2.5 7B in that same format. Generation speed drops off steeply with model size in the paper’s charts. Qwen 2 0.5B runs at roughly 46 tokens per second on the Orin’s CPU cores and roughly 33 on the Pi, while Phi 3.5 mini at 3.8B parameters sits at about 5 on the Pi, about 10 on the Orin CPU and about 15 on the Orin GPU. The authors also contradict Meta’s claim that Llama 3.2 is optimised for Arm processors, since the advantage they saw for Llama 3.2 1B showed up in GPU inference and not on either CPU. Memory is where the measurements turn hardest. Peak usage grows close to linearly with model size, reaching about 7.9 GB for Gemma 2 9B on the Pi, and it flattens out on the Orin above Phi 3.5 mini because under 7 GB of the nominal 8 GB is usable once the operating system and background processes take their more than 1 GB. Raising the context window to 1028 tokens pushed the largest model into constant page faults on the CPU and made the Orin crash and restart during GPU inference. First-run load times ran from 5 seconds to over 200 seconds, and for Yi 1.5 6B and up the faster 4-bit format raised load times by 17 to 53 times, because those models get evicted from memory and re-read from the microSD card for every inference. Scaled to the 18.84 Wh battery of a Samsung Galaxy S24 Ultra at full utilisation, the March version of the paper puts Gemma 2 9B at 193 inference runs on the Pi and 101 on the Orin GPU, against 3,928 and 4,280 runs for Qwen 2 0.5B, with no configuration lasting 7 hours. The setup was built to keep thermal throttling out of the results rather than to observe it, so the authors fitted the Pi with an add-on heatsink after it overheated in early experiments, ran the fans on both boards at full speed and left 30 seconds between runs. Quantisation cost measurable quality. Perplexity, which tracks how uncertain a model is about its next token, rose against the 16-bit weights for every model and both 4-bit formats and rose most sharply for the smallest models, while the downstream benchmarks showed almost no accuracy drop. The paper is on arXiv and under review at an ACM journal. Update, November 20, 2025. The authors revised the paper after validating their power measurements against external hardware, which raised the measured consumption. The battery estimate for Gemma 2 9B falls from 193 runs to 75 on the Pi and from 101 to 73 on the Orin GPU, and no configuration now reaches 3.5 hours at full utilisation. The speed, memory and quality results are unchanged. Source: https://arxiv.org/abs/2503.09114 Read the article: https://llmobile.kavents.com/ticker/on-device-inference-tradeoffs/ResearchBenchmarksllama.cppQuantisationNvidiahttps://llmobile.kavents.com/ticker/mobillm-side-tuning/https://llmobile.kavents.com/ticker/mobillm-side-tuning/Thu, 27 Feb 2025 08:58:02 +0100Researchers at Pengcheng Laboratory, the University of Houston, Stevens Institute of Technology, Keio University and the University of Waterloo posted MobiLLM on arXiv on February 27, 2025, a way to fine-tune a language model on a single device while a server does the training work. Their OPT-1.3B table puts the device-side memory at 4.50 GB, against 14.57 GB for LoRA, 13.37 GB for BitFit, 10.80 GB for the LST side-tuning baseline and 20.90 GB for full fine-tuning, all at a batch size of 16 and a sequence length of 256. The device is an NVIDIA Jetson Xavier NX whose GPU can use 4.6 GB of the board’s 8 GB of shared RAM, so MobiLLM is the only method in the authors’ comparison that fits a billion-parameter fine-tune onto it. For OPT-350M the same table gives 1.64 GB against 6.70 GB for LoRA and 7.91 GB for full fine-tuning, which is 0.01 GB more than running that model for inference on the device. Side tuning puts the two halves of training in two places. The device keeps the pre-trained backbone frozen and runs only the forward pass through it, while the server holds a small trainable side network built from adapter modules, each one a projection down to a narrow dimension, a non-linear step and a projection back up. The adapters sit parallel to the backbone rather than inside it, so gradients never travel back through the frozen model, and the authors trace the saving to the device storing neither the per-layer activations that backpropagation would need nor any optimizer state. It is a different bargain from splitting one model across several nearby devices, as Confidant does, which the authors argue needs a stable set of peers and leaves no single device holding a model it can run on its own. What crosses the network is one-way and happens only during the forward pass. The device quantises the intermediate activations of selected backbone blocks, by default to 4 bits, and sends them to the server along with the labels for the batch and metadata such as batch indices, and nothing travels back while training runs. Raw samples stay on the device, which is the authors’ privacy argument, though the labels do leave it, and the paper states that the activation outputs of each block distort the input embeddings enough to make the original samples hard to infer, without reporting a reconstruction attack that tests this. Quantising those activations cuts what has to be sent per iteration for OPT-1.3B from 400 MB to 100.2 MB, at a GLUE average of 81.3 in the NF4 format against 81.5 unquantised. The authors put the speed gain at 1.8 times to a target test accuracy on the Xavier and 2.3 times on a CPU-only Huawei MateBook laptop with a Core i5-13500H, both measured against LoRA, over a Wi-Fi 5 link to a server with an NVIDIA A100. Their link-rate table for OPT-350M is the more direct measurement, putting one iteration at a batch size of 16 at 7.48 s over 60 Mbps against 8.4 s for training on the device alone, and at 9.8 s when the uplink drops to 10 Mbps, where they say transmission becomes the bottleneck. Accuracy is the cost they name, with MobiLLM averaging 81.3 across the eight GLUE tasks on OPT-1.3B against 83.6 for full fine-tuning and 83.0 for LoRA, and 78.8 against 80.7 on OPT-350M. Fine-tuning this way still needs a reachable server, and the authors point at 5G and 6G base stations and home Wi-Fi hubs as where one would sit rather than testing either. They name two next steps, quantising the frozen backbone weights themselves, for which a side experiment puts device-side memory at 2.751 GB for OPT-1.3B at INT4 precision against a 2 percent accuracy drop and at 1.176 GB for OPT-350M, and filtering which activations get sent at all to cut the upload further. The paper is posted under the arXiv non-exclusive distribution license rather than a Creative Commons one, and it names no code release. Source: https://arxiv.org/abs/2502.20421 Read the article: https://llmobile.kavents.com/ticker/mobillm-side-tuning/ResearchMemoryQuantisationNvidiahttps://llmobile.kavents.com/ticker/genai-at-the-edge-survey/https://llmobile.kavents.com/ticker/genai-at-the-edge-survey/Wed, 19 Feb 2025 21:43:00 +0100Seven researchers at Johns Hopkins University and Duke University published a survey of generative AI on edge devices on February 19, 2025. Most of the accelerator hardware it collects has not been built. Its table of 12 accelerators for language and diffusion models covers 8 simulator studies against 4 fabricated chips, and the highest peak energy efficiency in it is 74.34 TOPS/W, from a 28nm computing-in-memory chip designed for Stable Diffusion. Compression is the part the survey treats as settled enough to quantify, and it attributes every figure to the work it reviews. It reports unstructured pruning reaching up to 60 percent sparsity on large models and a sparsity-aware inference system turning that into a 3 times speedup, while structured pruning that removes whole components gives about 2 times with little quality loss. Quantisation has been taken to 1 bit per weight in work such as BitNet, and the survey names what is still open there as holding generation quality under extreme compression and finding efficient ways to train quantised models on the edge devices themselves rather than only to run them. The frameworks chapter is where the survey names hardware a reader can buy. It cites a measurement that applying NVIDIA’s TensorRT to transformer models on a Jetson Xavier cuts latency by over 60 percent with negligible accuracy loss, and points to MELTing Point, a benchmark suite that tracks language model energy use and memory footprint across smartphones and Jetson boards. The authors state that no dedicated survey of generative AI at the edge existed before theirs, close by naming model personalisation and security across distributed edge nodes as the challenges that persist, and publish on arXiv under a Creative Commons Attribution 4.0 license. Diagram: Navardi and colleagues, from the survey. Source: https://arxiv.org/abs/2502.15816 Read the article: https://llmobile.kavents.com/ticker/genai-at-the-edge-survey/ResearchChipsQuantisationNPUhttps://llmobile.kavents.com/ticker/pareto-q/https://llmobile.kavents.com/ticker/pareto-q/Tue, 04 Feb 2025 19:59:00 +0100Meta published ParetoQ on February 4, 2025, a training framework that puts 1-bit, 1.58-bit, 2-bit, 3-bit and 4-bit quantisation through the same recipe so the bit widths can be compared against each other rather than against methods built for one width. The result the authors draw from it is that 1.58-bit, 2-bit and 3-bit land close together and generally beat 4-bit on accuracy against model size, while 1-bit drops away. On LLaMA-3 8B the paper’s table gives an average of 71.2 across five commonsense tasks at 2 bits and 69.0 at 1.58 bits against 74.6 for the full-precision model, then 64.1 at 1 bit. That ranking only holds when size is counted in bytes rather than parameters, which is how a phone’s memory budget works. The authors measure each model by its effective quantised size, every weight counted at its own bit width plus the embeddings at theirs, and compare across that axis. Their example is a 2-bit MobileLLM-1B scoring 61.0 on an eight-task average against 59.2 for a 4-bit MobileLLM-600M, 1.8 points higher at a smaller effective size, and the paper reports the same pattern across LLaMA models up to 8B. The paper also describes where the training behaviour changes. At 3 bits and above the fine-tuned weights stay within 10 to 20 percent of the full-precision weights they started from and recover most of the accuracy after about 10B tokens, while at 2 bits and below they move by roughly 40 percent and need around 30B tokens, which the authors read as the network rebuilding its representations instead of nudging them. On that basis they argue against training ternary models from scratch, the route taken by BitNet b1.58 and Spectra, and put their ternary 600M model at 58.7 on a six-task average, against 57.5 for the ternary 3B model of the 1-bit Era and 58.2 for Spectra’s ternary 3.9B, both computed from the same table. For speed the authors wrote a 2-bit CPU kernel and timed five MobileLLM models on an Apple M1 MacBook Pro with 32 GB of memory using 6 threads, 5 prompt tokens and 122 generated tokens, and report the 2-bit curve sitting above the 4-bit one on accuracy against decoding speed without naming a factor. The one speedup they put a number on is on an H100 NVL GPU, where their 2-bit kernel built on CUTLASS runs 4.14 times faster than FP16 and 1.24 times faster than the 4-bit Machete kernel from vLLM at a 16384 by 16384 weight shape. At 1B, 3B and 8B the authors say the in-kernel conversion overhead cancels that gain and 4-bit gives the better speed-accuracy trade-off. Hardware is also why the paper favours 2-bit over ternary despite the two sitting together on its accuracy-size front. Storing three values needs either sparsity above 90 percent to pay for the indexing or packing that complicates the matrix multiply, and the authors note that some implementations store ternary weights as 2-bit signed integers anyway, which gives up the storage saving. They write that broad adoption will need “community-wide efforts, such as INT2 support in NVIDIA tensor cores”, and the paper itself names no weight release. Update, May 30, 2025. Meta published the ParetoQ checkpoints on Hugging Face, five MobileLLM sizes from 125M to 1.5B, each as a BF16 model plus 1-bit, 1.58-bit, 2-bit, 3-bit and 4-bit versions, 30 repositories in all. They sit behind a gated form under Meta’s FAIR Noncommercial Research License, which permits research use only. The training code had gone up on GitHub on March 3, 2025. Source: https://arxiv.org/abs/2502.02631 Read the article: https://llmobile.kavents.com/ticker/pareto-q/MetaResearchQuantisationMemoryBenchmarkshttps://llmobile.kavents.com/ticker/slms-can-still-pack-a-punch/https://llmobile.kavents.com/ticker/slms-can-still-pack-a-punch/Fri, 03 Jan 2025 20:53:57 +0100Shreyas Subramanian, Vikram Elango and Mecit Gungor of Amazon surveyed about 160 papers on small language models on January 3, 2025, covering general-purpose models under 8B parameters plus a few exceptions up to 13B. Their central claim is that some of these models reach the performance of models 10 to 100 times their parameter count, a ratio the authors call effective size and anchor to PaLM at 540B parameters. In their table, Phi-1 at 1.3B parameters is credited with an effective size of 175B on code benchmarks and Phi-2 at 2.7B with 70B. The survey is clear about where that stops. Its appendix prints completions from 14 models for the prompt “The diagonals of a rectangle”, where a 33M TinyStories model answers “and they were so beautiful” and a 125M GPT Neo model loops on its own words, while TinyLlama at 1.1B and Phi 1.5 at 1.3B give the correct definition. The authors conclude that several billion parameters are the minimum for usable general-purpose responses, and they caution that every effective size they report comes from each paper’s best reported benchmark, so results on other benchmarks can be much lower. The clearest wins they list are task-specific and stay inside one domain, with WizardMath at 7B beating Llama 2 70B on maths benchmarks and Code Llama at 7B beating it on coding benchmarks. On what closes the gap, the survey puts training data first. It credits Phi-1 with 50.6 percent on HumanEval at 1.3B parameters trained on textbook-quality synthetic data, and TinyGSM, where a 1.3B generator paired with a 1.3B verifier scored 81.5 on the GSM8K maths set against 77.4 for the GPT-3.5 model that wrote its training data. Distillation covers the training side, with Orca 2 at 13B leading Llama 2 70B by an average of 3.23 points after learning step-by-step explanations from a teacher model rather than just its answers. After training, the survey groups quantisation and pruning, citing SmoothQuant for 8-bit weights and activations at up to 1.56 times the speed and half the memory, and Sheared-LLaMA for cutting Llama 2 7B down to 1.3B and 2.7B using 3 percent of the compute that training those sizes from scratch would need. The authors name resource-constrained deployment as the main motivation for the whole field, listing mobile devices, edge computing and regions with limited connectivity, and point at SmolLM at 135M, 360M and 1.7B parameters as a family built to run locally on phones and laptops. They note that extreme quantisation and sparse training already put Llama inference on CPUs within reach, and that 8-bit Adam training of a 7B model fits on a single 24 GB Nvidia RTX 4090. The paper is on arXiv under a CC BY-NC-ND 4.0 license, and its authors maintain a leaderboard of small models on Hugging Face Spaces. Source: https://arxiv.org/abs/2501.05465 Read the article: https://llmobile.kavents.com/ticker/slms-can-still-pack-a-punch/ResearchBenchmarksDistillationQuantisationPhihttps://llmobile.kavents.com/ticker/bluelm-v/https://llmobile.kavents.com/ticker/bluelm-v/Sat, 16 Nov 2024 01:14:51 +0100Researchers at CUHK MMLab and vivo AI Lab published BlueLM-V-3B on November 16, 2024, a vision-language model designed together with the code that runs it on a phone. The authors deployed it on the NPU of a MediaTek Dimensity 9300 in a vivo X100 and report a peak memory requirement of 2.2 GB and a generation speed of 24.4 tok/s with the language model weights quantised to 4-bit. The model pairs a 2.7B in-house BlueLM language model with a 400M SigLIP vision encoder that takes 384 by 384 image tiles. The authors report an average score of 66.1 on OpenCompass, a suite of eight multimodal benchmarks, which they state is the highest among models of 4B parameters or fewer, ahead of the 8B models MiniCPM-V-2.6 and InternVL2-8B. Most of the speed comes from how the model decides what resolution to feed the encoder. Dynamic resolution schemes pick a grid of tiles to fit an image, and the authors show that existing choices enlarge pictures a lot, with LLaVA-NeXT resizing and padding a 394 by 390 image to 768 by 768, four times the area, and InternVL 1.5 stretching a 380 by 76 image to 1920 by 384, twenty-five times the area. More pixels mean more tiles and more image tokens, so BlueLM-V-3B uses what the authors call relaxed aspect ratio matching, which tolerates a slightly smaller effective resolution when it wastes less of the canvas. Both examples then land on a single 1:1 tile, and across the 665k-sample LLaVA training set the method picks a smaller grid than InternVL 1.5 in 523k cases. The rest is shaped around what an NPU is bad at. The authors encode image tiles in batches of four at a time, which they measured as the fastest setting, and run the SigLIP convolution layer on the CPU in parallel with the transformer blocks on the NPU to hide 200 ms of latency. A downsampler merges every 2 by 2 patch into one token, cutting 729 tokens per tile to 196, and the prefill runs in chunks of 128 tokens rather than all at once. Precision is mixed, with INT4 for the language model weights, INT8 for the vision encoder and projector weights and for the KV cache, and INT16 and FP16 activations, which the authors say keeps accuracy stable because activations are more sensitive to quantisation than weights. Against the 8B MiniCPM-V 2.5 running on the CPU of the same chip through llama.cpp, the authors report 2.53 s to load and encode a 768 by 1536 image and 2.7 s of prefill, versus 4.0 s and 13.9 s, at 24.4 tok/s against 4.9 tok/s. The paper describes no public release of weights or code, and the authors say future work will target a broader range of mobile devices. Source: https://arxiv.org/abs/2411.10640 Read the article: https://llmobile.kavents.com/ticker/bluelm-v/MediaTekNPUQuantisationResearchBenchmarkshttps://llmobile.kavents.com/ticker/phonelm/https://llmobile.kavents.com/ticker/phonelm/Thu, 07 Nov 2024 03:19:00 +0100Researchers at Beijing University of Posts and Telecommunications published PhoneLM on November 7, 2024, a small language model family in 0.5B and 1.5B sizes built in an unusual order. The team first searched transformer configurations for the fastest inference on a specific phone, then pre-trained the configuration that won, instead of choosing a shape for capability and optimising speed after the fact. PhoneLM-1.5B decodes at 58 tok/s on the CPU of a Xiaomi 14 with a Qualcomm Snapdragon 8 Gen 3. The search varied the layer count from 15 to 25, the ratio of intermediate to hidden size between 2 and 5 and the attention type, and the paper lists measured throughput for seven candidate 1.5B shapes on that phone. The winner has 19 layers and a hidden size of 2,560, a hidden size to layer ratio of 134.7 against the 54.6 to 88.6 the authors observe in recent small models. It uses ReLU rather than the SiLU activation most current small models adopt, which the authors argue is cheaper on NPUs built for integer arithmetic and leaves more zeros in the feed-forward output for sparse-computation tricks. The paper notes that only the 1.5B model went through the search. Across seven zero-shot tasks including HellaSwag, PIQA, BoolQ and both ARC sets, the benchmark table puts PhoneLM-1.5B at an average of 67.31%, ahead of SmolLM-1.7B at 66.49% and StableLM 2 1.6B at 66.34% and behind DCLM-1.4B at 68.33% and Gemma 2 2B at 71.31%. PhoneLM-0.5B averages 60.43% against 60.17% for SmolLM-360M and 56.46% for OpenELM-450M. The authors measure PhoneLM-1.5B decoding 1.2 times faster than StableLM 2 1.6B and 1.6 times faster than SmolLM 1.7B on the same handset. The two models saw 1.1T and 1.5T training tokens, all of them from open datasets such as DCLM-baseline, StarCoderData and Dolma. The group ran every device measurement through its own inference engine mllm, with the linear and embedding weights quantised to 4-bit and four threads pinned to the phone’s performance cores. On the NPU it goes through Qualcomm’s QNN framework, where the authors report prompt processing for PhoneLM-1.5B at 654 tok/s on the Xiaomi 14 against 602 tok/s for Qwen 2.5 1.5B. The 4-bit mllm build the group publishes is a 1.41 GB file. The paper states no peak memory figure. An Android demo app on top of mllm runs PhoneLM-1.5B locally for chat and for Android intent invocation, so a user who asks to be woken at 8 o’clock has an alarm set through the system’s own alarm intent. That variant, PhoneLM-1.5B-Call, was fine-tuned with LoRA on DroidCall, 10,000 synthetic function-calling examples covering alarms, timers, email drafts and searches, and the paper places it above GPT-4o-mini, which scores 71.0% on the same test set. Weights are on Hugging Face under Apache 2.0 and the data preparation, training, fine-tuning and evaluation code is on GitHub under GPL-2.0, alongside the list of open datasets used at each stage. Source: https://arxiv.org/abs/2411.05046 Read the article: https://llmobile.kavents.com/ticker/phonelm/ResearchOpen weightsAndroidNPUAgentshttps://llmobile.kavents.com/ticker/small-model-training-bottlenecks/https://llmobile.kavents.com/ticker/small-model-training-bottlenecks/Fri, 25 Oct 2024 12:30:00 +0200Seven Apple researchers measured what it costs to train language models up to 2B parameters on rented Nvidia GPUs and found the most expensive card was the worst buy. The H100-80GB returned the fewest tokens per dollar of the three cards tested at every one of their four model sizes, 100M, 500M, 1B and 2B, while costing 2.53 times as much per hour as the A100-40GB. The price scale comes from normalising published hourly rates at Lambda Labs and Google Cloud, with the A100-40GB set at 1.00 and the A100-80GB landing at 1.46. The same comparison shows cost efficiency falling as GPUs are added, with a single card the best or near-best point at all four model sizes. For the 1B model, tokens per dollar drop from about 17,000 on one GPU to about 10,300 on 64. FlashAttention mattered more at small scale than at large. Reading the authors’ first figure, the 100M model gained roughly three times the tokens per dollar over vanilla attention, while at 2B the gain was under two times. The authors put this down to the cost of attention being quadratic in context length, so it dominates once the hidden dimension shrinks and the small models become bound by data movement between CPU and GPU and between GPUs. Vanilla attention also ran out of memory at a global batch size of 1024 on the 1B and 2B models, where FlashAttention did not. Two habits carried over from large-model training did not pay off. Plain Distributed Data Parallel, which keeps a full copy of the model on every GPU, beat both sharded FSDP variants for the smaller models because it moves less data, and FSDP only pulled ahead at 2B, where it kept training at a global batch size that made DDP run out of memory. Filling the cards was not cost-optimal either, since the batch size figure peaks around a global batch size of 128 for the 100M model, 32 for 500M, 16 for 1B and 8 for 2B, in every case well short of what the memory allowed. These are training runs on datacentre GPUs, not anything running on a phone, and the authors frame the work as guidance for labs with small budgets rather than for deployment. They name their own limits, running each setup at least three times for only 10 training steps, reporting normalised price ratios rather than absolute dollars, using no gradient accumulation, and leaving GPU utilisation and memory bandwidth measurements to future work. They also assume that settings such as the learning rate can be retuned to converge at whichever hardware configuration turns out cheapest. The 8-page paper is free to read on arXiv under the site’s non-exclusive distribution licence. Source: https://arxiv.org/abs/2410.19456 Read the article: https://llmobile.kavents.com/ticker/small-model-training-bottlenecks/AppleNvidiaResearchBenchmarksPyTorchhttps://llmobile.kavents.com/ticker/on-device-collaborative-lm/https://llmobile.kavents.com/ticker/on-device-collaborative-lm/Sat, 21 Sep 2024 00:34:00 +0200Four researchers at EPFL published CoMiGS on September 20, 2024, a scheme in which many devices fine-tune a language model together, with some of the added model parts shared across everyone and the rest kept on the device that trained them. In a simulation of four users, each holding Wikipedia articles in a different language, the authors measure test perplexity at 47.19 with GPT-2 124M as the base model, against 54.38 when each user trains alone on local data and 58.80 for federated averaging, which pools every user’s update into one shared model. Perplexity measures how surprised a model is by the next token, so lower is better. The paper appeared at ICML 2025. Nobody retrains the base model. Each device trains small LoRA adapters on top of pretrained weights that stay frozen, and CoMiGS sorts those adapters into two kinds. Generalist adapters are averaged across all devices at every communication round, specialist adapters and a small router never leave the device, and the router decides per token which adapters to use. The authors train that router against a separate validation set rather than the training data, and report that in the first layer of the network common function words such as “and”, “a” and “the” go mostly to the generalists, while for the users trained on maths and programming text the domain-specific terms in the last layer go mostly to the specialists. Nothing here ran on a phone. The appendix states that every experiment except the centralised baseline ran on a single NVIDIA A100-SXM4-40GB, with four simulated users holding between roughly 400,000 and 1 million training tokens each, drawn from multilingual Wikipedia, SlimPajama, AG News and Common Corpus, and averaged over three seeds. The base models are GPT-2 124M and Llama 3.2 1B. Results in the paper’s table are mixed rather than uniform, with CoMiGS ahead of both baselines on multilingual Wikipedia, SlimPajama and Common Corpus, and behind federated averaging on AG News, at 33.53 against 31.84 with GPT-2 and 16.31 against 15.86 with Llama 3.2 1B, which is the one setup where the test data does not follow the training distribution. The second set of experiments gives users different numbers of adapters, two to four per device with one of them always a generalist, which is how the method is meant to cope with devices of unequal capacity. Against HetLoRA and FlexLoRA, two methods that vary the LoRA rank per device instead, the authors report lower perplexity on every in-distribution task, for example 18.68 against 21.41 and 24.63 on Common Corpus with Llama 3.2 1B. They put the overhead of their extra router at 1.25 percent more computation per forward pass and 1.25 percent more memory, with communication per round halved against federated averaging because only the generalist weights are sent. The paper names its own limits. The authors write that differential privacy on the aggregated generalist weights is something they “do not pursue here”, that robustness against attackers is beyond the scope of the work and that their aggregation carries no guarantee against adversarial parties, and that there is a limit to how much the generalists can prevent overfitting when the local task is easy. They also note that fine-tuning Llama 3.2 1B on multilingual Wikipedia or SlimPajama produced negligible improvements, which they attribute to overlap with its pre-training data, and that is why the Llama runs use Common Corpus instead. The code is on GitHub under the Apache 2.0 licence. Diagram: Fan et al., Figure 2 of the paper, licensed CC BY 4.0. The router and the specialists stay on the device. Source: https://arxiv.org/abs/2409.13931 Read the article: https://llmobile.kavents.com/ticker/on-device-collaborative-lm/Mixture of expertsResearchOpen sourceLlamahttps://llmobile.kavents.com/ticker/elastic-on-device-llm-service/https://llmobile.kavents.com/ticker/elastic-on-device-llm-service/Sun, 08 Sep 2024 08:32:08 +0200Researchers at Peking University and Beijing University of Posts and Telecommunications published ElastiLM on September 8, 2024, a phone-side language model service that resizes itself for every request. Running LLaMA-7B on a Redmi K60 Champion Edition with a Snapdragon 8 Gen 2, the authors measure 0.31 seconds to switch from one sub-model to another, against 8.3 seconds for swapping in a separately pretrained model of the right size and 6.2 seconds for the pruning method LLMPruner. The setup the paper assumes is one model held by the operating system that every app calls, the way Android AICore serves apps like Gboard smart reply. Those callers want different things from it, and the authors put an always-on voice assistant that needs an answer almost immediately next to a background screen-event recorder that can wait. ElastiLM takes a latency budget along with each request and then picks how much of the model to run and how much of the prompt to keep, in steps of 10 percent, rather than running one fixed model at one fixed speed. Accuracy is what pays for the speed. Cutting the prompt drops words the model might have needed and pruning removes whole attention heads and MLP neurons, and the split matters as much as the total, since the authors show one science question that half the prompt with an 80 percent sub-model answers correctly while 80 percent of the prompt with a half-size sub-model gets wrong. Preparing a model costs 68.3 GPU hours on a server with NVIDIA A40s, which the authors price at about $100 of rented time and put at 21.7 to 68.2 hours more than the pruning baselines. The service runs on four big CPU cores, and the authors state that as of August 2024 no mobile GPU or NPU could run ElastiLM or its baselines end to end, so their accelerator figures are projected from profiled latency rather than measured on a device. Across the Redmi K60 Champion, a Mi 14 and a Redmi K70 Pro, the last two on Snapdragon 8 Gen 3, the authors report ElastiLM answering more requests correctly than seven baselines by 6.60 to 14.83 percent in absolute accuracy, 10.45 percent on average, on traces of 600 requests drawn from six datasets. Peak memory for LLaMA-7B sits at 15 to 17 GB, level with the baselines, while keeping a dedicated model per latency budget would take 29.3 GB or 76.3 GB and runs out of memory on all three phones, which carry 19 GB to 24 GB of RAM. Weight-only quantisation to 8 bits costs almost no accuracy and 4 bits about 3 percent absolute, by the authors’ measurement. The paper appeared at MobiCom 2025 in Hong Kong in November 2025 and is on arXiv under a CC BY 4.0 license. Charts: Yin et al. Online overhead measured on a Redmi K60 Champion Edition, with the dashed line marking 1 percent of average time to first token. Source: https://arxiv.org/abs/2409.09071 Read the article: https://llmobile.kavents.com/ticker/elastic-on-device-llm-service/ResearchAndroidQualcommLlamahttps://llmobile.kavents.com/ticker/olmoe/https://llmobile.kavents.com/ticker/olmoe/Tue, 03 Sep 2024 19:08:20 +0200Ai2 published OLMoE-1B-7B on September 3, 2024, a language model with 6.9B total parameters of which only 1.3B run on any given token. It is a sparse mixture of experts, meaning each layer holds 64 small feedforward networks and a small router picks 8 of them per token, so the rest of the model stays idle. Compute per token lands near that of a dense 1B model, while all 6.9B parameters still have to be held in memory. Ai2 pretrained the model on 5 trillion tokens and reports in the paper that it leads every open model in its active-parameter class. On MMLU the paper puts OLMoE-1B-7B at 54.1, against 48.5 for DCLM-1B, 33.6 for TinyLlama-1B and 32.1 for Ai2’s own dense OLMo-1B. The paper states the model matches or outperforms the dense OLMo-7B at the end of pretraining while using less than half the training FLOPs, and that it beats Llama2-7B on MMLU, scored at 46.2, despite needing roughly 6 to 7 times less compute per forward pass. The instruction-tuned version is reported above Llama2-13B-Chat and DeepSeekMoE-16B. For the training-cost claim the authors ran a controlled comparison, training a 1.3B dense model and a 1.3B active, 6.9B total mixture of experts on 128 H100 GPUs for 130B tokens each. Ai2 reports the sparse model reaching the dense model’s final quality with about 3 times fewer tokens and FLOPs, but only about 2 times faster in wall-clock time, because the larger weight set dropped throughput to 23,600 tokens per second per GPU against 37,500 for the dense run. The full pretraining run used 256 H100 GPUs for around 10 days. Ai2 shipped an iOS app on February 11, 2025 that keeps the whole model on the device, so prompts and responses never reach a server and generation keeps working in airplane mode. Ai2 names the iPhone 15 Pro and 15 Pro Max, the four iPhone 16 models and M-series iPads going back to the 2021 iPad Pro, and states that the first version needs an iOS device with 8 GB of memory. The app runs a 4-bit quantised build of the January 2025 instruction-tuned model, which Ai2 measures at 41 tokens per second on average on an iPhone 16 Pro, on a stack Ai2 built with GenUI on top of llama.cpp. Weights, the training data mix, the training code and the run logs are all published under Apache 2.0, together with 244 intermediate checkpoints. The comparison table in the paper lists OLMoE as the only mixture-of-experts release that opens all four, with most of the other models in the table offering a single checkpoint and no logs. The app is in the App Store and its source code is on GitHub. Screenshot: Ai2. Update, April 14, 2025. Ai2 kept working on the app after the February launch and added macOS support on March 7, 2025 through Mac Catalyst, Apple’s compatibility layer that runs the same iOS code natively on a Mac, together with a counter that shows how many tokens a reply used. A later version reworked the panel that reports generation metrics, and the most recent release, from April 14, 2025, renamed the app to Ai2 OLMoE. Nothing has shipped in the repository since. Source: https://arxiv.org/abs/2409.02060 Read the article: https://llmobile.kavents.com/ticker/olmoe/Ai2Mixture of expertsiOSOpen weightsResearchhttps://llmobile.kavents.com/ticker/spectra/https://llmobile.kavents.com/ticker/spectra/Wed, 17 Jul 2024 07:53:00 +0200Nolano AI, the University of Montreal and Mila published Spectra on July 17, 2024, a suite of 54 language models covering nine parameter counts from 99M to 3.9B, each trained on the same 300B tokens of SlimPajama. Every size exists as a half-precision model the authors call FloatLM, as GPTQ-quantised copies at 3, 4, 6 and 8 bits called QuantLMs, and as a TriLM, a model trained from scratch with every linear-layer weight restricted to -1, 0 or 1. The paper’s size table puts TriLM 3.9B at 10.76 billion bits, roughly 1.3 GB, against 63.83 billion bits for the half-precision model of the same parameter count, a factor of 5.9. That is the comparison the suite is built for, since what fits on a device is bits rather than parameters. Averaged over six commonsense and reasoning benchmarks, the paper’s tables put TriLM 3.9B at 60.7 against 61.4 for FloatLM 3.9B, level with the 4-bit QuantLM 3.9B at 60.7 which needs 20.59 billion bits. Set against the models nearest to it in size, TriLM 3.9B beats the half-precision 830M model, 60.7 against 53.3 at 13.34 billion bits, and the 3-bit QuantLM 2.4B, 60.7 against 54.7 at 10.95 billion bits. On LAMBADA it reaches 61.6 against 61.1 for FloatLM 3.9B, and on TriviaQA 21.3 against 21.5. Chart: Kaushal et al. Averages across Arc Easy, Arc Challenge, BoolQ, HellaSwag, PIQA and WinoGrande. During training a TriLM keeps floating-point latent weights and ternarises them on the fly in the forward pass, with one scale value per weight matrix shard and gradients passed through by a straight-through estimator, so only the ternary states and the scales are needed at inference. Embeddings and the output head stay in half precision across the suite. The authors position TriLM against BitNet b1.58, which also trains ternary weights from scratch, and say they normalise once before each linear layer instead of normalising, scaling and quantising activations to 8 bits four to seven times per transformer block; their own replication of the BitNet architecture at 1.1B, trained on 100B tokens, scores below TriLM 1.1B on the same benchmarks. The paper names the weaknesses it measured. At 1.1B parameters the ternary model does not reach its half-precision counterpart, 51.6 against 54.9 on the six-benchmark average, and the 2.4B one still trails at 57.3 against 59.2, so the match with a half-precision model of the same parameter count holds at 3.9B only. On CrowsPairs, which measures how often a model prefers a stereotyped sentence, TriLM 3.9B lands at 65.4 percent against 64.7 for FloatLM 3.9B and 62.6 for the smaller FloatLM 830M, so the ternary model stereotypes more than the half-precision model closest to it in bits. No model in the suite beats random guessing on TruthfulQA, and while TriLM 3.9B holds up on Penn Treebank and LAMBADA perplexity, it stays behind FloatLM 3.9B on web corpora including Dolma, C4 and RefinedWeb, which the authors read as weaker memorisation of training data. The weights and more than 500 intermediate checkpoints are on Hugging Face under Apache 2.0. The TriLM checkpoints are published unpacked to FP16, which the model cards say makes them compatible with ordinary half-precision matrix multiplication and leaves the released files at full half-precision size. The authors list smartphones, laptops and automobiles among the resource-constrained devices they see for ternary models, and end by calling for work on the toxicity, stereotyping and web-corpus gaps they measured. Source: https://arxiv.org/abs/2407.12327 Read the article: https://llmobile.kavents.com/ticker/spectra/ResearchQuantisationMemoryOpen weightsBenchmarkshttps://llmobile.kavents.com/ticker/qwen2/https://llmobile.kavents.com/ticker/qwen2/Mon, 15 Jul 2024 14:35:00 +0200Alibaba’s Qwen Team published the Qwen2 Technical Report on July 15, 2024, covering five models from 0.5B to 72B parameters. The report states that the two smallest, Qwen2-0.5B and Qwen2-1.5B, are designed for easy deployment on portable devices such as smartphones, earphones and smart glasses, while the larger sizes target GPUs. Qwen2-1.5B scores 56.5 on MMLU, a multiple-choice knowledge test, against 52.7 for Microsoft’s Phi-2, which the same table credits with about twice the non-embedding parameters. Both small sizes tie the input embedding to the output layer, so a single weight matrix does both jobs and the shared vocabulary of 151,646 tokens is stored once rather than twice. The report puts non-embedding parameters at 0.3B for Qwen2-0.5B and 1.2B for Qwen2-1.5B. All five models use grouped query attention, where several query heads share one set of key and value heads, with 14 query heads over 2 key-value heads at 0.5B and 12 over 2 at 1.5B, which the authors say lowers the key-value cache size per token against Qwen1.5 and with it the memory footprint during long-context inference. Alibaba trained Qwen2-0.5B on a 12 trillion token dataset and the other dense models on a 7 trillion token one, reporting that the larger corpus did not pay off at the bigger sizes. For the instruction-tuned versions the report gives Qwen2-1.5B-Instruct 52.4 on MMLU, 61.6 on the GSM8K grade-school maths set and 29.0 on IFEval, which checks whether a model obeys explicit formatting instructions, against 37.9, 40.1 and 20.0 for Qwen2-0.5B-Instruct. The authors attribute the gain over the similarly sized Qwen1.5 models mainly to that data scaling. Context length differs across the family. The authors tested retrieval from long documents with Needle in a Haystack and report that the two small models handle 32K tokens, Qwen2-57B-A14B-Instruct 64K, and the 7B and 72B instruction-tuned models 128K with the YARN extension applied. The pre-training data covers approximately 30 languages, among them English, Chinese, Spanish, French, German, Arabic, Russian, Korean, Japanese, Thai and Vietnamese. Alibaba put the weights on Hugging Face and ModelScope. The Qwen2 blog post of June 7, 2024 states that Qwen2-0.5B, 1.5B, 7B and 57B-A14B carry the Apache 2.0 license while Qwen2-72B and its instruction-tuned variant stay on Alibaba’s own Qianwen License, and it names MLX, llama.cpp, Ollama and LM Studio among the projects that run the models locally. Source: https://arxiv.org/abs/2407.10671 Read the article: https://llmobile.kavents.com/ticker/qwen2/AlibabaQwenOpen weightsResearchWearableshttps://llmobile.kavents.com/ticker/fast-on-device-llm-inference-with-npus/https://llmobile.kavents.com/ticker/fast-on-device-llm-inference-with-npus/Mon, 08 Jul 2024 14:20:00 +0200Researchers at Peking University and the Beijing University of Posts and Telecommunications published llm.npu on July 8, 2024, an inference engine that runs the prefill stage of a language model on a phone’s NPU. They tested it on a Redmi K70 Pro with a Snapdragon 8 Gen 3 and a Redmi K60 Pro with a Snapdragon 8 Gen 2, using Qwen1.5-1.8B, Gemma-2B, Phi-2 2.7B, LLaMA2-7B and Mistral-7B. On 1024-token prompts on the K70 Pro the authors report prefill 7.3x to 38.4x faster than the CPU engines MNN and llama.cpp and 1.27x to 43.6x faster than the GPU engines TFLite and MLC-LLM, and for Qwen1.5-1.8B they measure 1,106 tok/s against 151 tok/s for MNN on the CPU and 57 tok/s for llama.cpp on the CPU. Energy was measured on the K60 Pro, the only device they could root, where prefill used 1.85x to 59.5x less energy than the same baselines. Chart: Xu et al., published under CC BY 4.0. The prompts come from the LongBench 2WikiMultihopQA dataset. Prefill is the stage where the model reads the whole prompt before it writes its first token, and the paper puts it at 88.3% to 98.8% of total latency on a mobile CPU for UI automation, chat summaries and context-aware generation, and at 54.2% to 91.7% on a mobile GPU. A mobile NPU is fast at integer matrix multiplication but expects fixed tensor shapes, while prompts arrive at any length, so every new length means rebuilding the graph. Quantised activations also contain outliers, single values far outside the usual range, and the accurate remedy is per-group quantisation, which mobile NPUs cannot execute directly and which the authors measure at up to 10.7x overhead when it is split into sub-tensor multiplications. llm.npu answers both by cutting a prompt into fixed 256-token chunks that run on pre-built graphs, sharing the operators that do not depend on prompt length so the memory cost drops by up to 4x, pulling the outlier values out to the CPU or GPU to run in parallel, and scheduling transformer blocks across the processors out of order. Accuracy stays within 1% of FP16 on LAMBADA, HellaSwag, WinoGrande, OpenBookQA and MMLU. Across three application workloads, UI automation, context-aware email replies and chat summaries, the authors measure end-to-end latency 1.4x to 32.8x lower than the baselines, with decoding still running unoptimised on the CPU. The system is built on the group’s own mllm inference framework and Qualcomm’s QNN SDK, in about 10,000 lines of C, C++ and assembly, and the code is public in the mllm repository under an MIT license. The paper appeared at ASPLOS 2025 in Rotterdam in spring 2025. PowerInfer-v2, the one comparison system that also uses the NPU for prefill, is not open source, so the authors took its figures from its paper and report a 3.28x to 5.6x speedup over it. Source: https://arxiv.org/abs/2407.05858 Read the article: https://llmobile.kavents.com/ticker/fast-on-device-llm-inference-with-npus/NPUQualcommAndroidQuantisationResearchhttps://llmobile.kavents.com/ticker/mobileclip/https://llmobile.kavents.com/ticker/mobileclip/Fri, 21 Jun 2024 02:15:00 +0200Five Apple researchers presented MobileCLIP at CVPR 2024 on June 20, a family of image-text models built for phones. They exported each model with Core ML Tools 7.0 and timed it at batch size 1 on an iPhone 12 Pro Max running iOS 17.0.3, and their Table 7 gives the smallest variant, MobileCLIP-S0, 1.5 ms for the image encoder and 1.6 ms for the text encoder at 67.8% zero-shot accuracy on ImageNet. The same authors measure OpenAI’s ViT-B/16 CLIP on that phone at 11.5 ms and 3.3 ms for 68.3%, and both models average 58.1 across the 38 datasets of the DataComp evaluation suite. Multi-modal reinforced training, the method the paper introduces, moves the expensive part of distillation out of the training loop and into the dataset. Apple reinforces the DataComp dataset once, writing five synthetic captions per image with the CoCa model in OpenCLIP, storing the parameters of strong random image augmentations, 30 per image for the 12M subset and 10 for the billion-sample set, and storing the embeddings that two ViT-L/14 teacher models produce for those augmented images and for the real and synthetic captions. Training afterwards reads the captions and the teacher embeddings from disk, so the captioning model and the teachers never run again however many models are trained on the set. The paper puts numbers on what that saves. One epoch over the 12.8M samples of the reinforced set on a single node of 8 A100-80GB GPUs takes 1.3 hours, the same as an epoch on the unreinforced data, against 21.1 hours when the captioning model and the teacher ensemble run during training, according to Table 4d. The cost moves to storage, and Table 4c puts DataComp-1B at 90 TB against 140 TB for the reinforced DataCompDR-1B, with the 12M subset going from 0.9 TB to 1.9 TB. MobileCLIP comes in four sizes, and Table 7 lists image and text encoder parameters separately. S0 pairs an 11.4M-parameter image encoder with a 42.4M-parameter text encoder, S1 uses 21.5M and 63.4M at 2.5 ms and 3.3 ms, S2 uses 35.7M and 63.4M at 3.6 ms and 3.3 ms, and MobileCLIP-B keeps a standard ViT-B/16 image encoder at 86.3M and 10.4 ms. Zero-shot ImageNet accuracy across the four runs 67.8%, 72.6%, 74.4% and 76.8%, and a version of B trained on 39B seen samples instead of 13B reaches 77.2%. All three small variants replace the vision transformer with a convolution-transformer hybrid derived from Apple’s FastViT, and S0 also swaps the text encoder for one that mixes 1-D convolutions with four self-attention layers, which the authors report as 5% smaller and 15.8% faster than the 12-layer transformer text encoder that S1 and S2 keep. Apple released the code, the weights and the reinforced data at apple/ml-mobileclip, with checkpoints for all four variants and the longer-trained B, and with DataCompDR-1B and DataCompDR-12M on Hugging Face as synthetic captions, augmentation parameters and teacher embeddings keyed to DataComp images rather than as the images themselves. The repository lists the code under MIT, the models under Apple’s own machine learning research model terms, and the data under CC BY-NC-ND. Apple showed zero-shot scene classification running in real time on an iPhone in the CVPR exhibit hall, and added an iOS app for the same task to the repository in November 2024. Source: https://openaccess.thecvf.com/content/CVPR2024/html/Vasu_MobileCLIP_Fast_Image-Text_Models_through_Multi-Modal_Reinforced_Training_CVPR_2024_paper.html Read the article: https://llmobile.kavents.com/ticker/mobileclip/AppleiPhoneResearchBenchmarksOpen weightshttps://llmobile.kavents.com/ticker/llms-on-mobile-measurements/https://llmobile.kavents.com/ticker/llms-on-mobile-measurements/Tue, 11 Jun 2024 15:00:00 +0200Five researchers at Beijing University of Posts and Telecommunications measured language model inference on four Android phones and presented the results at the EdgeFM workshop of MobiSys 2024, deploying 22 models from 0.5B to 7B parameters through llama.cpp. For the 18 models they charted at 4-bit quantisation, accuracy for most of them sat between 55 and 60 percent, averaged across Winogrande, HellaSwag and MMLU, and latency came out at about 200 ms per token averaged over the four devices. The phones were a Xiaomi 14 with a Snapdragon 8 Gen 3 and 16 GB of RAM, a Pixel 7 Pro with a Tensor G2 and 12 GB, a Xiaomi 12S with a Snapdragon 8 Gen 1+ and 12 GB, and a Meizu 18 Pro with a Snapdragon 888 and 8 GB. Most of the 7B models needed about 4 GB of memory, which the authors put at half or a third of a current phone’s RAM. Their memory leaderboard has Bloom-7B highest, at 1.27 times the footprint of MPT-7B at the same parameter count, and Qwen1.5 0.5B lowest at under 1 GB. Memory use did not move with input or output length in their runs, with Baichuan holding at 4.83 GB whether the prompt was 10 or 50 tokens long and whether 1 or 1000 tokens were generated, which they trace to llama.cpp allocating one large pool at startup rather than sizing it per phase. Newer chips helped the two inference phases unevenly. Against the Snapdragon 888, the authors measured the Snapdragon 8 Gen 3 raising prefill throughput on LLaMA 2 by 81% but decode throughput by only 43%, with the Snapdragon 8 Gen 1+ at 45% and 34%. Decode is the phase that emits one token at a time and so governs how fast text appears on screen, and there the gap between the 8 Gen 3 and the 8 Gen 1+ was 7%. Continuous prefill runs also pushed the Xiaomi 12S, Xiaomi 14 and Meizu 18 Pro into thermal throttling, with per-token times drifting during a test series. The processor comparison went against the GPU in most cases. In the engine test llama.cpp on the 8 Gen 3 CPU prefilled 1.63 times faster than MLC LLM on the same phone’s GPU, 10.3 tokens per second against 6.3, while MLC on the GPU decoded faster at 9.8 against 6.6. On the mid-range Snapdragon 7 Gen 2 the CPU was ahead of the GPU in both phases, 4.59 and 4.65 against 3.3 and 3.5. The authors put this down to mobile GPUs being built for rendering rather than general-purpose compute, and to the extra cost of moving data. Quantisation cost less than the hardware choices did. Testing eight bit-widths on LLaMA 2, the authors report 4-bit as the best compromise, giving up 0.6% average accuracy against the 32-bit model while cutting model size 8 times, where 2-bit halves the size again for four times the accuracy loss. Code and results are published as TinyLLMLeaderBoard on GitHub, and the six-page paper is free to read in the ACM Digital Library. Source: https://dl.acm.org/doi/10.1145/3662006.3662059 Read the article: https://llmobile.kavents.com/ticker/llms-on-mobile-measurements/BenchmarksResearchllama.cppQualcommAndroidhttps://llmobile.kavents.com/ticker/powerinfer-2/https://llmobile.kavents.com/ticker/powerinfer-2/Mon, 10 Jun 2024 16:01:21 +0200Researchers at the IPADS lab of Shanghai Jiao Tong University published PowerInfer-2 on June 10, 2024, an inference framework for running language models that exceed a phone’s memory. On a OnePlus 12, a Snapdragon 8 Gen 3 device with 24 GB of DRAM of which 19 GB can be occupied by an app, the authors measure TurboSparse-Mixtral-47B decoding at 11.68 tokens per second, and describe it as the first system to serve a 47B model on a smartphone. In the same configuration they report 3.12 times the speed of LLMFlash and 21.2 times the speed of llama.cpp. LLMFlash here is their own reimplementation of Apple’s LLM in a flash techniques inside llama.cpp, which they built because that work was never released as code. PowerInfer-2 splits each feedforward layer into neuron clusters, small groups of neurons that tend to activate together, and treats a cluster rather than a whole matrix as the unit of work. Frequently activated clusters go to the phone’s NPU for dense arithmetic, while rarely activated ones go to the CPU, which copes better with irregular sparse work. Weights that are not already in an in-memory cache are read from UFS flash on demand, and those reads are overlapped with computation so that a core can work on one cluster while the next is still arriving. For prompt processing the system loads whole layers with sequential reads, which the authors measure at 3 times the throughput of random reads, and prefetches the next layer while the current one computes. The authors vary how much memory the app may use from 7 GB to 19 GB with the 47B model. At 7 GB the cache holds only 1.8 percent of the feedforward weights, so nearly every neuron has to be fetched from flash for every token and decoding falls to 2.13 tokens per second, which is still 1.84 times LLMFlash. The 47B model activates only about 3B parameters per token thanks to its mixture-of-experts structure and added sparsity, which is why it decodes at roughly the pace of a 7B model in the same test. Streaming weights off storage to fit a large model into a small budget is the same bet Edge0 makes at the level of whole experts rather than individual neurons. Chart: Xue et al. LLMFlash is the authors' own reimplementation of Apple's method. On OpenCompass at 4-bit weights, the authors put PowerInfer-2 at an average of 78.38 across Arc-Challenge, Arc-Easy, MMLU and GSM8K for Qwen2 7B against 79.25 for llama.cpp, and at 68.35 against 70.12 for Bamboo-7B, with Qualcomm’s QNN engine at 56.93 and 63.26. Token latency is uneven, and for the 47B model with half the feedforward weights offloaded they report a mean of 99.76 ms per token and a 99th percentile of 140.56 ms, tracking a cache miss rate that averages 3.5 percent but reaches 18.9 percent at the 99th percentile. Prompt processing for Bamboo-7B runs at 404.6 tokens per second with half the weights offloaded, against over 700 tokens per second with everything in memory. They measure energy at 0.257 joules per token against 0.373 for QNN and 0.672 for llama.cpp. PowerInfer-2 uses privileged system calls such as memory locking, so the authors built and ran it on rooted Android phones, and they state that nothing in the design requires kernel changes and that it should port to iOS. It also needs models with predictable sparsity, which is why the group trained sparsified TurboSparse versions of Mistral and Mixtral, and on an unmodified SiLU-based Mistral-7B the gain over LLMFlash drops to 2.4 times against 4.6 times for the ReLU-based Bamboo-7B. The system is 12,000 lines of C++ added to the group’s PowerInfer engine for PCs, which is MIT-licensed, but the announcement there links the paper and the model weights rather than a release of the PowerInfer-2 code. The paper was last revised in December 2024. Source: https://arxiv.org/abs/2406.06282 Read the article: https://llmobile.kavents.com/ticker/powerinfer-2/MemoryNPUAndroidQualcommResearchhttps://llmobile.kavents.com/ticker/mobile-foundation-model-as-firmware/https://llmobile.kavents.com/ticker/mobile-foundation-model-as-firmware/Wed, 29 May 2024 15:00:00 +0200Researchers at Beijing University of Posts and Telecommunications proposed that a phone ship one shared multimodal model instead of letting every app bundle its own, in a paper published in the ACM MobiCom 2024 proceedings on May 29, 2024. The operating system and the hardware co-manage that model like firmware, unchangeable by apps or by the OS itself, exposed to applications as a system service, and each app reaches it through a small adapter fine-tuned offline for its own task. Their prototype, called M4, holds 9.2B parameters and needs 7.5 GB of peak memory, and the authors report it reaching accuracy comparable to purpose-built models on 85% of the 50 datasets in a benchmark they assembled from 38 mobile AI tasks across five input types. What the shared model replaces is one small model per app per task. The paper’s baselines are 50 task-specific models of 1M to 500M parameters each, one per dataset, against which M4’s adapters run from 1,000 to 10 million parameters, so each added task costs under 10 MB. Measured on an Nvidia Jetson Orin NX, 4-bit M4 needs 6.1 GB of storage to serve all 50 tasks against 15.2 GB for the 50 separate models, with the crossover at about 15 tasks, and 7.5 GB of peak memory against roughly five times that. The authors state that on a device with 12 GB of memory the 4-bit model plus all 50 adapters fits, where only 20 of the 50 task-specific models would. The prototype is slower than the models it replaces. On the Jetson Orin NX with 16 GB, the authors measured M4 averaging 18 times the inference latency of the task-specific models across the 50 tasks and 19 times the energy, 3.6 s against 0.2 s. On a Pixel 7 Pro CPU they measured an average of 6.8 s against 0.54 s, and their per-task breakdown puts image classification at 2.10 s and question answering at 6.34 s to the first token and 0.24 s per token after it. They state that M4 cannot currently run on a stock smartphone GPU or NPU at all, because those processors lack support for the operators it uses. The NPU numbers in the paper are a projection rather than a measurement. The authors estimate that M4 on an NPU would average 0.48 s and 1.3 J, under the 0.54 s and 2.9 J they measured for task-specific models on the Pixel 7 Pro CPU, but they derive that by applying the CPU-to-NPU ratio they observed for task-specific models, not by running M4 on an NPU. Their case for a simpler accelerator rests on a separate Pixel 7 Pro measurement, where they converted 110 downloaded models to TensorFlow Lite and only 8% ran entirely on the NPU, those gaining a median speedup above 20 times over the CPU. M4 itself uses 39 operator types against the 156 that the 50 task-specific models need between them. The authors name their own limits. They write that the accuracy results come from an A100 and the Jetson board rather than from phones, that M4 underperforms task-specific models on some tasks including translation, and that a prototype assembled from off-the-shelf pre-trained models is “still highly inefficient in terms of accuracy and model parameter size”. Its backbone is Meta’s LLaMA-7B at 8-bit, with encoders taken from ImageBind and Whisper, and they note that adapters trained against one backbone stop working when the backbone is upgraded, so the design still needs a stable interface between the two. Code and benchmark are published at github.com/UbiquitousLearning/MobileFM, and the paper carries ACM copyright rather than an open license. Source: https://dl.acm.org/doi/10.1145/3636534.3649361 Read the article: https://llmobile.kavents.com/ticker/mobile-foundation-model-as-firmware/ResearchBenchmarksNPUPixelLlamahttps://llmobile.kavents.com/ticker/openelm/https://llmobile.kavents.com/ticker/openelm/Tue, 23 Apr 2024 01:12:00 +0200Apple posted OpenELM to arXiv on April 22, 2024, a family of four language models at 270M, 450M, 1.1B and 3B parameters, each also released in an instruction-tuned version. The name stands for Open-source Efficient Language Models, and the efficiency comes from layer-wise scaling, which spreads the parameter budget unevenly across the transformer’s layers instead of giving every layer the same shape. Table 1 of the paper gives the 1.1B model an average of 45.93% across the OpenLLM leaderboard tasks against 43.57% for OLMo 1.2B, a gap the authors state as 2.36%, reached on 1.5T pre-training tokens against OLMo’s 3.0T. Layer-wise scaling varies the number of attention heads and the width of the feed-forward network from layer to layer, keeping layers near the input narrower and widening them towards the output, so a layer late in the stack holds more parameters than an early one. Apple trained the four variants for 350k iterations with its CoreNet library on public data only, drawing on RefinedWeb, a deduplicated PILE, a subset of RedPajama and a subset of Dolma v1.6, about 1.8T tokens in total. Text is filtered and tokenised on the fly rather than pre-tokenised, using the same tokeniser as Llama. The paper benchmarks the models on an Apple MacBook Pro with an M2 Max chip and 64 GiB of RAM running macOS 14.4.1, through a port of OpenELM to MLX v0.10.0, and separately on a Linux workstation with an Intel i9-13900KF and an NVIDIA RTX 4090. On the MacBook in BFloat16, Apple measures the 270M model generating 212.40 tok/s and the 3B model 33.96 tok/s, with the 4-bit versions at 256.35 and 60.33 tok/s. On the CUDA machine OpenELM 1.08B is slower than OLMo 1.18B, at 92.15 tok/s against 203.40, which the authors attribute to a naive RMSNorm implementation and to OpenELM having 113 normalisation layers where OLMo has 33. Apple presents the release as the complete framework rather than weights and inference code, covering the training and evaluation code for public datasets, the training logs, multiple checkpoints, the pre-training configurations and the code that converts a model to MLX for inference and fine-tuning on Apple devices, with pre-converted bf16 and 4-bit checkpoints for all four sizes. The eight models are on Hugging Face and the code is in CoreNet. Both the model cards and CoreNet name the terms as the Apple sample code license, which is Apple’s own license and not one of the standard open source licenses. Source: https://arxiv.org/abs/2404.14619 Read the article: https://llmobile.kavents.com/ticker/openelm/AppleApple SiliconResearchOpen weightsBenchmarkshttps://llmobile.kavents.com/ticker/octopus-v3/https://llmobile.kavents.com/ticker/octopus-v3/Wed, 17 Apr 2024 17:07:06 +0200Wei Chen and Zhiyuan Li posted Octopus v3 on April 17, 2024, a technical report on a multimodal agent model that takes an image alongside a text query and answers with a function call. The model has fewer than 1B parameters, handles English and Chinese in one set of weights, and is built to run on edge devices. The paper lists both authors at Stanford University and gives a Nexa AI address for correspondence. The model encodes each supported function as a single functional token in its vocabulary, learned the way a language model learns an unfamiliar word from the contexts it appears in, so the output is one token for the action plus its arguments and needs no parser to read. Images run through a CLIP-based encoder, which the authors say they chose after comparing encoding methods. Training runs in stages, with the language model and the image encoder trained apart, then merged and aligned, then taught the functional tokens using the method from Octopus v2, and finally tuned by reinforcement learning with another large language model acting as the reward model. The evaluation consists of 10 smartphone functions converted into functional tokens and ten worked examples, each printing the model’s output next to the output of GPT-4V feeding GPT-4, which the authors prompted with all the function descriptions and a few examples. The authors write that their model performs comparably to that pair on the selected functions. The report carries no accuracy rate, no success rate and no benchmark table, so the comparison rests on the ten printed generations rather than on a measured score. The report states that the model runs on devices as constrained as a Raspberry Pi and that it makes converting a Raspberry Pi into something like a Rabbit R1 or a Humane AI Pin practical with an on-device model instead of a cloud service. It names no Raspberry Pi board, no phone and no other hardware it was run on, and gives no latency, no memory footprint and no throughput, so the edge deployment is asserted rather than shown. The authors do name one cost themselves, writing that vision input may introduce considerable latency and that they were optimising inference speed. Weights and inference code were available only on application when the report went up, which describes the model as still under testing and for research purposes only, and the authors state that the functional token is licensed. The project page that hosted the demo videos no longer resolves. The authors list audio and video as the modalities they want to support next. Source: https://arxiv.org/abs/2404.11459 Read the article: https://llmobile.kavents.com/ticker/octopus-v3/Nexa AIAgentsResearchhttps://llmobile.kavents.com/ticker/octopus-v2/https://llmobile.kavents.com/ticker/octopus-v2/Tue, 02 Apr 2024 11:01:00 +0200Wei Chen and Zhiyuan Li published Octopus v2 on April 2, 2024, a 2B function-calling model fine-tuned from Google’s Gemma 2B that turns a spoken or typed request into an Android API call. On their own evaluation set built from 20 Android APIs, the authors report 99.524% accuracy and 0.38 seconds per call for Octopus v2, against 98.571% and 1.02 seconds for GPT-4. Both authors give Stanford University as their affiliation on the paper and publish the weights under the name Nexa AI. The distinguishing idea is what the authors call a functional token. Each of the 20 APIs gets its own special token added to the tokenizer and to the model’s output head, so picking a function becomes a single choice among 20 token candidates instead of writing the function name out as text and hoping every piece of it lands correctly. Since the model learned during fine-tuning what each of those tokens stands for, the function descriptions no longer have to sit in the prompt, which the paper puts at over 95% less context. Retrieval-augmented function calling processes roughly 1,000 tokens per call in their comparison, while Octopus v2 sees the query alone, typically under 30 tokens. Shorter context is also where the speed comes from. The authors measure Llama-7B with retrieval-augmented function calling at 68.095% accuracy and 13.46 seconds per call on a single NVIDIA A100, and Octopus v2 at 0.38 seconds under the same settings on the same A100, a 35-fold difference. The phone figure is a separate measurement. With the model quantised and the state for the fixed prompt prefix precomputed, they report a function call finishing in 1.1 to 1.7 seconds for queries of 20 to 30 tokens on what the paper calls a standard Android phone, without naming the device, and they estimate 37 times more calls per iPhone battery charge than a 7B model with retrieval. The authors name limits of their own result. Training with LoRA instead of the full model drops accuracy to 99.048%, and cutting the training set from 1,000 to 100 generated examples per API drops it to 98.095%, which they still put forward as the low end of a usable range. The set of callable functions is fixed at training time, since every API has to exist as a token before the model can emit it, and they price the data generation at 0.0224 USD per 1,000 examples per API. They also note that the GPT-3.5 and GPT-4 latencies were taken through OpenAI’s API on March 18, 2024, and may reflect API traffic or hardware allocation at that moment. Nexa AI released the weights on Hugging Face under a non-commercial CC BY-NC 4.0 license, on top of Google’s Gemma 2B. The model card said at the time that the team was still preparing the code for an open-source release. Source: https://arxiv.org/abs/2404.01744 Read the article: https://llmobile.kavents.com/ticker/octopus-v2/Nexa AIAgentsAndroidGemmaResearchhttps://llmobile.kavents.com/ticker/octopus-function-calling/https://llmobile.kavents.com/ticker/octopus-function-calling/Tue, 02 Apr 2024 03:29:28 +0200Wei Chen, Zhiyuan Li and Mingyuan Ma of Stanford University and Harvard University posted Octopus on April 2, 2024, four open models fine-tuned to pick the right software API for a request and fill in its arguments. On a benchmark the authors built themselves, they report their fine-tuned Gemma 2B answering 93 percent of calls correctly against 96 percent for GPT-4 and 50 percent for GPT-3.5, with the CodeLlama 7B and Gemma 7B versions reaching 97 percent once an output filter is switched on. The title says on-device, but nothing in the paper runs on a device. The authors fine-tuned the models on A100 80GB GPUs and queried GPT-4 and GPT-3.5 through OpenAI’s API, so every number was measured server-side. The device claim is a single sentence stating that models of this size can already be deployed on mobile, pointing at MLC LLM, and the paper names no phone, no chip and no latency, memory or throughput figure. The training set starts from roughly 30,000 of the most used APIs on RapidAPI Hub, which GPT-4 cut to about 20,000 by dropping entries with missing arguments or descriptions that did not match their parameters. The authors rewrote each API as a Python function signature with a docstring, had GPT-4 write five answerable queries per API and an equal share of unanswerable ones, and let GPT-4 verify the result, ending at about 150,000 examples as described in the paper. Vector search over the function descriptions pulled in near-identical functions as distractors, and training worked through them from easy to hard. Base models were CodeLlama 7B, Gemma 2B and 7B and Stable Code 3B, all tuned with LoRA at rank 16 and 8-bit quantisation, using 90 GPU hours for CodeLlama 7B and Gemma 7B, 60 for Stable Code 3B and 30 for Gemma 2B. The output filter the authors call a conditional mask blocks, at each generated token, anything that would break the expected format, so a function name has to come from the known list and an argument has to match its declared type. The authors state that it cannot be applied to GPT-4 or GPT-3.5 because the OpenAI API does not return the underlying token scores, and that GPT-3.5 was not counted wrong for formatting mistakes. The paper states that the training dataset and the fine-tuned models will be open sourced soon, and names no repository or date. The preprint went up under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license. Source: https://arxiv.org/abs/2404.01549 Read the article: https://llmobile.kavents.com/ticker/octopus-function-calling/Nexa AIResearchAgentsGemmaBenchmarkshttps://llmobile.kavents.com/ticker/on-device-llmaas-context/https://llmobile.kavents.com/ticker/on-device-llmaas-context/Mon, 18 Mar 2024 15:03:23 +0100Researchers at Peking University, the Beijing University of Posts and Telecommunications and Tsinghua University published LLMS on March 18, 2024, an operating system service that keeps a separate conversation context for every app calling one shared on-device language model. Serving eight active contexts of Llama2-7B on a Jetson Orin NX, the authors measure 0.27 seconds to switch to another app’s context. They report switches up to 20 times and on average 9.7 times faster than a baseline that swaps the same chunks but quantises every one to 8 bits, modelled on the server system vLLM. The premise is one model held by the operating system instead of a copy inside each app, the way Android AICore serves Google apps, with every caller keeping its own KV cache, the stored attention state that lets a model continue a conversation without rereading it. On a Xiaomi 14 running llama.cpp, the authors measure one Llama2-7B context at the full 4k-token window at 2.02 GB, against 3.92 GB for the 4-bit weights, so a few apps holding contexts outweigh the model they all share. Letting the phone’s low-memory killer reclaim that memory means rebuilding the context by running the text through the model again, which the authors put at 22.92 seconds and 94.57 J on the same phone, close to the 90.04 J they measure for one minute of YouTube. What moves to storage is the KV cache, cut into chunks of 16 tokens, while the prompt and output text stay resident. The service quantises each chunk to 8, 4 or 2 bits according to how much attention its tokens draw from the rest of the context, holding a 50 percent average set by the system, then refills a context by reading some chunks from disk while recomputing others from the stored text and overlapping the two. The authors report twice the compression ratio of static quantisation at negligible accuracy loss, where compressing every chunk alike to 4 bits or 2 bits costs up to 59 percent and 99 percent of accuracy. Each of the three techniques carries part of that 0.27 seconds, which the authors break down as 0.42 seconds without the per-chunk compression, 0.62 seconds without the eviction order and 1.62 seconds without the read-and-recompute pipeline. The prototype is 3.5k lines of Python and C++ on Hugging Face Transformers and mllm, run on a Jetson Orin NX, a Jetson TX2 and a Xiaomi 14 with a Snapdragon 8 Gen 3, each with 8 GB of RAM, using Llama2-7B and OPT-6.7B over 72-hour traces synthesised from six datasets, and the authors state that it leaves token generation speed within 5 percent of running without it. The paper is on arXiv under a CC BY 4.0 license, and the same four authors published the work at SenSys 2026 in Saint-Malo in May 2026 as An Efficient Context Management System for On-Device LLMaaS. Charts: Yin et al. Measured on a Xiaomi 14, with the memory breakdown taken on llama.cpp. Source: https://arxiv.org/abs/2403.11805 Read the article: https://llmobile.kavents.com/ticker/on-device-llmaas-context/ResearchMemoryQuantisationLlamaAndroidhttps://llmobile.kavents.com/ticker/merino-iot-language-models/https://llmobile.kavents.com/ticker/merino-iot-language-models/Wed, 28 Feb 2024 04:20:00 +0100Researchers at the University of Central Florida and the University of Technology Sydney published MeRino on February 28, 2024, a method that draws up a transformer architecture to fit a given device’s limits on parameters, arithmetic and latency, instead of shrinking an existing model to fit. Their largest design, MeRino-64M, reaches the same 0.408 average accuracy across twelve zero-shot tasks as Meta’s OPT-350M while carrying 64M parameters against 331M. The authors time it at 114 ms against 559 ms for a 128-token input on an NVIDIA Jetson Nano, or 4.9 times faster. The search scores a candidate architecture without ever training it. Each candidate shape, meaning how many blocks it has and how wide and how deep each block is, gets a single number that the authors compute from the dimensions of its attention and feedforward matrices and treat as a measure of how much information the network can represent, with a second term that penalises shapes which are deep and narrow, because those train badly. An evolutionary algorithm mutates block widths and depths for 100,000 rounds, discards everything that breaks the budget, keeps the highest-scoring survivors and returns the best one. A precomputed lookup table stands in for the matrix decomposition the score would otherwise need, and the authors put its average error at 0.03 percent. Because the score is arithmetic over an architecture’s dimensions rather than a forward pass, the search runs on the target board itself. The paper puts one MeRino search at 0.05 hours and 0.75 Wh on the Jetson Nano’s CPU, against 1.2 hours and 300 Wh for the TE-NAS architecture search on an NVIDIA GTX 1080Ti. Every device measurement in the paper comes from that Jetson Nano developer board, taken at batch size 1 and sequence length 128 and averaged over 16 runs, and the authors report no measurements on phones or tablets. Training the designed models is separate and conventional work, at 600,000 steps on 8 NVIDIA H100 GPUs. The same tables show MeRino-64M behind OPT-350M on language modelling, at 22.47 against 18.51 perplexity on WikiText-2 and 27.06 against 23.08 on Penn Treebank, where lower is better. The smallest design, MeRino-52M, matches Pythia-70M at 0.383 average zero-shot accuracy and 48 ms against 95 ms. The authors state that the weights and evaluation for Meta’s MobileLLM were not released when they submitted, so they do not compare against it, and they report that 8-bit weight quantisation leaves accuracy almost unchanged while 4-bit costs about 1 percent. The work was accepted at AAAI 2025 and the paper names no code release. Source: https://arxiv.org/abs/2403.07921 Read the article: https://llmobile.kavents.com/ticker/merino-iot-language-models/ResearchBenchmarksNvidiaMemoryhttps://llmobile.kavents.com/ticker/stable-lm-2-zephyr/https://llmobile.kavents.com/ticker/stable-lm-2-zephyr/Tue, 27 Feb 2024 20:00:00 +0100Stability AI published the Stable LM 2 1.6B technical report on February 27, 2024, describing a decoder-only model of 1.6B parameters trained from scratch on roughly 2 trillion tokens of public data. The mix is multilingual by design, with web text in German, Spanish, French, Italian, Dutch and Portuguese alongside English, and the report states the finished model handles all seven. The model has 24 layers and a context length of 4096 tokens, which the authors list as one of the things they want to extend. In the report’s multilingual table, which averages translated versions of four English benchmarks and a next-word prediction set across those seven languages, the base model scores 40.5, above Google’s Gemma 2B at 39.8 with 2.5B parameters and Microsoft’s Phi-2 at 34.6 with 2.7B. The chat variant reaches 41.5 and Stability’s own 3B model 41.7. The authors state they used no multilingual data during fine-tuning, so the language coverage comes out of pre-training alone. On the six English tasks of the Hugging Face Open LLM Leaderboard, the same report’s table puts the base model at an average of 45.3, behind Alibaba’s Qwen 1.5 1.8B at 46.6 and Gemma 2B at 46.5. The instruction-tuned StableLM 2 Zephyr 1.6B averages 49.7, the highest of the sub-2B entries in that table, ahead of Phi-1.5 at 47.7 and behind the 2.7B Phi-2 at 61.3. The report’s abstract calls Stable LM 2 1.6B the state-of-the-art open model under 2B parameters at the time of publication. Stability built the Zephyr variant in three stages. It fine-tuned the base model on seven public instruction datasets, 826,938 samples after dropping conversations longer than eight turns, then aligned the result with direct preference optimisation, which trains on pairs of preferred and rejected answers, using UltraFeedback and Intel’s Orca Pairs and borrowing most hyperparameters from the recipe behind Hugging Face’s Zephyr 7B. A third stage taught the model who made it and what a language model cannot do, using 10,000 opening messages the base model wrote for itself. The variant scores 5.42 on MT-Bench, where a larger model judges answers to open-ended questions, above Qwen 1.5 1.8B Chat at 5.29 and Gemma 2B Instruct at 5.19 and below Stability’s own StableLM Zephyr 3B at 6.64. Chart: Bellagente et al., the authors' own figures. Stability put both sets of weights on Hugging Face along with quantised builds for llama.cpp, Apple’s MLX and Intel’s OpenVINO. The report measures 127 tok/s at 11 W for the 4-bit MLX build on an M2 Mac mini with 8 GB, against 71 tok/s at 6 W in FP16, and 99 tok/s for a 4-bit GGUF build on a 2023 MacBook Pro with 16 GB, while stating these figures are meant as a practical reference rather than rigorous benchmarking. Neither release is open source. The report says the model ships under a non-commercial licence, the Zephyr model card names the Stability AI Non-Commercial Research Community License and points commercial users to the company, and the base model card now carries the Stability AI Community License with commercial use running through a Stability membership. Source: https://arxiv.org/abs/2402.17834 Read the article: https://llmobile.kavents.com/ticker/stable-lm-2-zephyr/Stability AIResearchBenchmarksOpen weightsApple Siliconhttps://llmobile.kavents.com/ticker/bitnet-b1-58/https://llmobile.kavents.com/ticker/bitnet-b1-58/Tue, 27 Feb 2024 19:56:00 +0100Microsoft Research and the University of Chinese Academy of Sciences published BitNet b1.58 on February 27, 2024, a language model in which every single weight is one of three values, -1, 0 or 1. At 3B parameters the authors measure it at 2.22 GB of memory against 7.89 GB for a full-precision Llama-style model of the same size trained on the same number of tokens, a 3.55 times reduction, with a slightly lower perplexity of 9.91 against 10.04. Three possible values need log2(3) bits to store, which is roughly 1.58 and where the name comes from. What that buys is a change to the matrix multiplication that dominates the work of running a model. A normal model multiplies each weight by an activation in floating point and sums the products, but multiplying by -1, 0 or 1 is only a sign flip, a skip or a copy, so the paper states that the matrix multiplication of BitNet “only involves integer addition”. Activations stay at 8 bits. The models are trained from scratch at this precision rather than trained in full precision and squeezed down afterwards, which is how most low-bit models on phones are produced today. The authors trained BitNet b1.58 and their own reproduced Llama baseline on the same data for the same number of tokens at each size, and compare the two directly. They report that BitNet b1.58 starts to match the full-precision baseline in perplexity at 3B and on zero-shot end tasks from 3B as well, where it averages 50.2 against 49.7. The efficiency figures widen with size. Memory drops 2.60 times at 700M, 2.93 times at 1.3B and 3.55 times at 3B, and latency per output token falls 1.23 times, 1.67 times and 2.71 times across the same three sizes. For throughput the authors ran 70B models on two A100 cards and report a batch size of 176 against 16 and 2977 tokens per second against 333, an 8.9 times gain. They also calculate that on 7nm chips the arithmetic for matrix multiplication costs 71.4 times less energy, because integer addition is far cheaper than floating-point multiply-accumulate. The paper names no model release and its arXiv listing marks it as work in progress. The authors argue the approach suits CPUs, “which are the main processors used in edge and mobile devices”, and say the lower memory and energy cost is what would let such models run there. They close by calling for new hardware and systems designed specifically for 1-bit models. Update, April 16, 2025. Microsoft released BitNet b1.58 2B4T, a 2B model trained from scratch on 4 trillion tokens at ternary precision, which the team calls the first open-source native 1-bit model at that scale. The report puts its non-embedding memory at 0.4 GB against 1.4 GB for Gemma-3 1B and 2.6 GB for Qwen2.5 1.5B, measures CPU decoding at 29 ms per token on a 13th Gen Intel Core i7-13800H using 8 threads against 41 ms and 65 ms for the same two models, and estimates energy at 0.028 J against 0.186 J and 0.347 J. Across the benchmark suite it averages 54.19, above Llama 3.2 1B at 44.90 and just under Qwen2.5 1.5B at 55.23. The weights are on Hugging Face, with a custom CUDA kernel for GPUs and CPU inference through bitnet.cpp, the runtime Microsoft published in October 2024 and measured at 2.37 to 6.17 times faster than the baseline on x86 CPUs and 1.37 to 5.07 times faster on ARM. Source: https://arxiv.org/abs/2402.17764 Read the article: https://llmobile.kavents.com/ticker/bitnet-b1-58/MicrosoftResearchQuantisationMemoryChips