https://llmobile.kavents.com/categories/on-device/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/mediatek-dimensity-9600-pro/https://llmobile.kavents.com/ticker/mediatek-dimensity-9600-pro/Tue, 15 Sep 2026 00:00:00 +0200MediaTek announced the Dimensity 9600 Pro, a flagship smartphone chip built on a 2nm process. According to the company, its NPU 1090 supports on-device applications with models of up to 30B parameters. MediaTek reports 51% higher LLM prefill performance, the phase where the model reads the prompt, and 55% higher token generation per watt, measured on demo devices in its own labs. The chip pairs the NPU 1090 with a second-generation Super Efficient NPU, which MediaTek says cuts power consumption for always-on AI by 40%. The platform supports LPDDR6 memory and UFS 5.0 storage. In September 2025, MediaTek announced that it had completed the tape-out, the final design handoff to the fab, of a flagship chip on TSMC’s N2P 2nm process, with volume production expected in late 2026. The CPU uses a 2+3+3 layout of eight big cores, with two C2-Ultra cores at up to 4.55 GHz. MediaTek states up to 17% higher single-core and up to 15% higher multi-core performance over the previous generation, and up to 61% lower multi-core power consumption. The first smartphones with the Dimensity 9600 Pro and the related Dimensity 9600M are expected to launch this quarter, according to MediaTek. Source: https://www.mediatek.com/press-room/mediatek-dimensity-9600-pro-sets-new-standard-for-flagship-smartphone-chips Read the article: https://llmobile.kavents.com/ticker/mediatek-dimensity-9600-pro/ChipsAndroidSmartphoneMediaTekhttps://llmobile.kavents.com/ticker/arm-create-china-2026/https://llmobile.kavents.com/ticker/arm-create-china-2026/Fri, 11 Sep 2026 00:00:00 +0200Arm has published five developer takeaways from Arm Create, its developer events in Shanghai and Shenzhen. Two of them concern on-device AI. Arm says model choice starts with the workload and not with model size alone, and that developers should decide which parts of an application stay on the device, which run on nearby edge infrastructure and which need the cloud. The Shenzhen panel included Alibaba Qwen, ModelBest, Tencent Hunyuan and Ultralytics. In the Shanghai keynote, Shantu Roy, Arm’s VP of Developer Relations, discussed the Arm AI Portal. Arm says the portal lists models validated and optimized for Arm-based platforms, together with performance data for specific targets, code and deployment workflows. Coding agents can reach the same information through the Arm MCP Server. The recap shows the portal’s evaluation of Qwen3-TTS 0.6B Custom Voice, a multilingual streaming text-to-speech model from Alibaba, on a mobile CPU. The entry lists a vivo X300 with 8 CPU cores and 16 GB of memory, SME2, the XNNPACK and KleidiAI optimizations, FP16 weights and the LiteRT runtime. It reports a real-time factor of 1.2x against a baseline of 0.28x and a median end-to-end latency of 3,878 ms against 16,877 ms. Peak memory is 4,727 MB against 6,718 MB, and the evaluation uses the English subset of the MiniMaxAI TTS-Multilingual-Test-Set. Evaluation results in the Arm AI Portal, as shown in Arm's recap. Source: Arm. Arm also points to Arm CSS for Mobile 2, which combines the Arm C2 CPU Cluster with SME2 and the Mali G2-Ultra NX GPU. Arm says the platform supports new on-device AI experiences on mobile. The next Arm Create event moves to the US, and Arm has not given a date. Source: https://newsroom.arm.com/blog/takeaways-from-arm-create-china-2026 Read the article: https://llmobile.kavents.com/ticker/arm-create-china-2026/ChipsModelsAndroidTTShttps://llmobile.kavents.com/ticker/edge0-35b-iphone-demo/https://llmobile.kavents.com/ticker/edge0-35b-iphone-demo/Thu, 10 Sep 2026 16:08:02 +0200Samuel Zeng has released Edge0, an open-source framework for streaming mixture-of-experts inference, under Apache-2.0. In his launch post he reports a 35-billion-parameter model running on an iPhone at 1 to 2.5 GB of peak memory without a cloud connection. The repository describes three techniques. Expert weights are streamed from storage on demand, so peak memory is bounded by the active set. A trained prerouter predicts expert routing one step ahead so that loads overlap the forward pass, which the project credits with up to 59 percent higher throughput. Recover-LoRA pairs an int4-quantised frozen base with distilled LoRA adapters. Two tiers are published. edge0-35b has 40 layers and 256 experts, a checkpoint of roughly 23 GB and about 2.9 GiB of peak active memory. edge0-8b has 24 layers and 128 experts, a 4.2 GB checkpoint and about 1.0 GB peak. On a Mac mini M4 Pro with 24 GB, the project measures 14.9 to 17.7 tokens per second of decode for the 35B tier and 23.9 to 25.3 for the 8B tier, with prefill at 113 tokens per second cold and 140 warm for 35B. Quality loss against FP16 baselines is given as 3.9 points on average for the 35B tier and 2.8 points for the 8B tier. The repository states that the MLX backend runs on macOS with Apple Silicon, on M1 through M4, and requires Python 3.10 or newer. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Demo by Samuel Zeng. Watch original on X Source: https://x.com/SamuelZengML/status/2097861839287927139 Read the article: https://llmobile.kavents.com/ticker/edge0-35b-iphone-demo/iPhoneApple SiliconOpen weightsMixture of expertsMemoryhttps://llmobile.kavents.com/ticker/llama-cpp-hexagon-npu-snapdragon/https://llmobile.kavents.com/ticker/llama-cpp-hexagon-npu-snapdragon/Thu, 10 Sep 2026 11:00:00 +0200A user has reported results from llama.cpp’s Hexagon NPU backend on a OnePlus 12 with a Snapdragon 8 Gen 3. They cross-compiled llama.cpp on Ubuntu following the project’s Snapdragon documentation and ran it from Termux on the phone. The reported figures are 8 tokens per second of prompt processing and 4.5 of generation with gemma-3-12b-it-qat at Q4_0, and 20 tokens per second of prompt processing and 12.5 of generation with gemma-3-4b-it-qat at Q4_0. According to the report the speed is about the same as running on the same chip’s CPU, but the phone does not get hot. The report lists several limitations. The backend accepts only Q4_0, IQ4_NL, MXFP4, Q8_0 and F32 files, and does not support KV cache quantisation. It runs on chips as old as the Snapdragon 888, but the poster notes that only the 8 Gen 2 and newer carry the tensor module used for language models. The Hexagon NPU can address 4 GB, so a model plus KV cache beyond that has to be split across more than one NPU device: LD_LIBRARY_PATH=./lib:/vendor/lib64 \ ADSP_LIBRARY_PATH=./lib \ GGML_HEXAGON_NDEV=2 \ ./bin/llama-completion \ -m /sdcard/gguf/gemma-3-12b-it-qat-Q4_0.gguf \ -ngl 99 --device HTP0,HTP1 \ -sys 'You are a helpful AI assistant' For context the poster cites 34 INT8 TOPS and 76.8 GB/s of memory bandwidth for the 8 Gen 3 NPU, against 80 TOPS and 228 GB/s for Qualcomm’s X2 Elite Extreme and 248 TOPS and 936 GB/s for an Nvidia RTX 3090. They ask readers with a Snapdragon 8 Elite Gen 5 phone or an X2 Elite Extreme laptop to post their own numbers, since those parts are said to address more than 4 GB and would not need multiple devices. Source: https://www.reddit.com/r/LocalLLaMA/comments/1t0k6fj/running_llamacpp_on_snapdragon_hexagon_npu_seems/ Read the article: https://llmobile.kavents.com/ticker/llama-cpp-hexagon-npu-snapdragon/QualcommAndroidNPUllama.cppGemmahttps://llmobile.kavents.com/ticker/iphone-18-pro-a20-pro-neural-engine/https://llmobile.kavents.com/ticker/iphone-18-pro-a20-pro-neural-engine/Wed, 09 Sep 2026 19:00:00 +0200Apple has introduced the iPhone 18 Pro and iPhone 18 Pro Max with the A20 Pro chip. According to Apple, the chip has a new dual 16-core Neural Engine, 32 cores in total, which the company describes as double the AI processing power of A19 Pro. Apple says the Neural Engine accelerates on-device AI models and computational photography. Image: Apple. The A20 Pro also offers 50 percent more memory bandwidth than A19 Pro, and its 6-core CPU includes integrated Neural Accelerators. Apple positions the chip for “more advanced on-device AI workloads” alongside games. The phones ship with iOS 27, Apple Intelligence and the new Siri AI. Apple states that Apple Intelligence uses on-device processing together with Private Cloud Compute. Pre-orders start on 12 September, with availability from 18 September; prices start at $1,199 for the iPhone 18 Pro and $1,299 for the iPhone 18 Pro Max, both with 256 GB. Source: https://www.apple.com/newsroom/2026/09/apple-debuts-iphone-18-pro-and-iphone-18-pro-max/ Read the article: https://llmobile.kavents.com/ticker/iphone-18-pro-a20-pro-neural-engine/AppleiPhoneChipsNPUhttps://llmobile.kavents.com/ticker/arm-mali-g2-ultra-nx/https://llmobile.kavents.com/ticker/arm-mali-g2-ultra-nx/Tue, 08 Sep 2026 00:00:00 +0200Arm has introduced the Mali G2-Ultra NX, a smartphone GPU that places dedicated neural accelerators inside each shader core. The accelerators reuse the GPU’s memory system, coherent caches and control structures, support INT8 and INT16 processing and include hardware-accelerated optical flow for motion estimation. Arm calls it the first AI-native Mali GPU and positions the accelerators for neural graphics at 1 W. The GPU is part of the Arm CSS for Mobile 2 platform. The GPU reaches up to 24% higher benchmark performance than the previous generation and 14% higher performance in non-AI gaming workloads, Arm says. Its neural graphics features are Neural Super Sampling, which reconstructs a higher-resolution image from a lower-resolution render, Neural Frame Rate Upscaling, which generates intermediate frames, and Neural Super Sampling and Denoising, which combines upscaling with denoising for ray-traced scenes. Arm says frame rate upscaling supports up to 120 FPS for longer gaming sessions. In its Neural Dawn demo, Arm reports up to 4x higher performance efficiency and up to 70% lower external memory traffic. Arm's overview slide for the Mali G2-Ultra NX. Source: Arm. Arm's diagram of how Neural Frame Rate Upscaling builds an intermediate frame. Source: Arm. The new execution engine is the largest update to the Mali instruction set architecture in seven generations, Arm says, with up to 2x more registers per warp. The third-generation hardware ray tracing unit adds support for Opacity Micromaps, which handle complex transparent geometry. Arm reports up to 13% lower DRAM traffic on ray tracing benchmarks, a 30% higher frame rate and up to 70% less ray tracing work in a scene from Moku’s Central Garden. For developers, Arm offers the Arm Neural Graphics Development Kit with machine learning extensions for Vulkan, plug-ins for Unreal Engine, an SDK for custom engines and tools for profiling, training and model optimization. Arm names integrations with Tencent Games Central Tech’s Magic Dawn engine, Unity China’s Tuanjie Engine and Unreal Engine MegaLights. Keli Zhou, engine lead for Where Winds Meet, says the game will be among the first to bring Arm Neural Technology to players. Arm’s post names no launch dates for devices with the GPU. Source: https://newsroom.arm.com/blog/arm-mali-g2-ultra-nx-ai-native-mobile-graphics Read the article: https://llmobile.kavents.com/ticker/arm-mali-g2-ultra-nx/ChipsAndroidhttps://llmobile.kavents.com/ticker/arm-css-for-mobile-2/https://llmobile.kavents.com/ticker/arm-css-for-mobile-2/Tue, 08 Sep 2026 00:00:00 +0200Arm has introduced Arm CSS for Mobile 2, a compute platform for smartphone chips that combines the C2 CPU cluster, the Mali G2-Ultra NX GPU and the SI L2 system interconnect. The C2 cluster pairs C2-Ultra and C2-Pro CPUs with two SME2 units, the Scalable Matrix Extension 2 that speeds up matrix math for AI on the CPU. Arm says this doubles the SME2 capability of the previous-generation configuration and reports up to 1.7x performance across the latest AI models. The cluster delivers up to 15% higher single-thread performance, 15% faster web browsing, 12% faster app launch and 12% higher multi-thread performance, Arm reports. For AI, it cites a peak uplift of up to 70% in selected tasks. Its slide compares speech, personal memory retrieval and prefill, the phase where a model reads the prompt, against the C1-Ultra with SME2. In a representative agentic workflow covering speech processing, memory retrieval, reasoning, app execution and web browsing, the C2-Ultra with two SME2 units finishes 24% faster than the previous generation, according to Arm. Arm's own comparison of the C2-Ultra with the C1-Ultra, with the AI tasks measured against the C1-Ultra with SME2. Source: Arm. The example flagship configuration in Arm’s slides has two C2-Ultra and six C2-Pro cores. The Mali G2-Ultra NX GPU integrates neural accelerators into its shader cores and adds a new execution engine and a third-generation ray tracing unit for neural graphics. Arm says the SI L2 interconnect provides lower-latency access, higher bandwidth, coherency and quality-of-service controls for CPU, GPU and other resources working at the same time. Partners can use each component on its own or combine them with custom and third-party IP. Arm's slide for the C2-Ultra CPU with an example flagship cluster layout. Source: Arm. On the software side, Arm lists KleidiAI, its optimized libraries for Arm CPUs including SME2 paths, plus integrations with common AI frameworks. The Arm AI Portal offers validated models with performance and accuracy data, code examples and deployment resources, and the Arm MCP Server connects them to agentic development tools. vivo says it is bringing Arm Neural Technology to its latest flagship smartphones built on the platform, aimed at mobile gaming. Arm’s post names no launch dates for devices with CSS for Mobile 2. Source: https://newsroom.arm.com/blog/arm-css-for-mobile-2-and-c2-cpu-cluster Read the article: https://llmobile.kavents.com/ticker/arm-css-for-mobile-2/ChipsAndroidhttps://llmobile.kavents.com/ticker/minicpm5-2b/https://llmobile.kavents.com/ticker/minicpm5-2b/Mon, 07 Sep 2026 10:00:00 +0200OpenBMB has released MiniCPM5-2B, published on Hugging Face and dated 7 September 2026 in the project changelog. The model card describes a dense Transformer “built for on-device, local deployment, and resource-constrained scenarios”, with 2,516,756,480 parameters of which 1,981,982,720 are non-embedding. The architecture uses 42 layers with grouped-query attention at 16 query heads and 2 key-value heads, and supports a context length of 131,072 tokens. It is the second model in the MiniCPM5 series after MiniCPM5-1B, scaling the same recipe to a larger footprint. OpenBMB reports an average score of 53.9 across its evaluation dimensions, including 69.1 on LiveCodeBench, 86.5 on AIME 2025 and 68.1 on NoLiMa. The release is licensed under Apache-2.0 and ships in BF16, GGUF, GPTQ 4-bit, MLX for Apple Silicon and LiteRT-LM for mobile, plus a DSpark draft model for speculative decoding. Source: https://huggingface.co/openbmb/MiniCPM5-2B Read the article: https://llmobile.kavents.com/ticker/minicpm5-2b/OpenBMBOpen weightsQuantisationLiteRThttps://llmobile.kavents.com/ticker/gemini-nano-4-first-devices/https://llmobile.kavents.com/ticker/gemini-nano-4-first-devices/Thu, 27 Aug 2026 16:00:00 +0200Google’s ML Kit GenAI documentation now lists the first devices running nano-v4, 9to5Google reports. The list covers the Pixel 11, Pixel 11 Pro, Pixel 11 Pro XL and Pixel 11 Pro Fold, plus Samsung’s Galaxy Z Flip8, Galaxy Z Fold8 and Galaxy Z Fold8 Ultra. The same documentation sets Nano v3 or greater as the requirement for Gemini Intelligence, Google’s on-device feature set. According to the report, that requirement first appeared in May 2026, was removed, and has now been reinstated. Listed hardware requirements include 12 GB or more of RAM, a qualified flagship system-on-chip, five or more OS upgrades and six years of security support. Gemini Intelligence features named in the report include Rambler and Proactive Assistance on Pixel 11, and task automation across more than 40 apps on Samsung’s foldables. Source: https://9to5google.com/2026/08/27/gemini-intelligence-nano-4/ Read the article: https://llmobile.kavents.com/ticker/gemini-nano-4-first-devices/GoogleAndroidPixelSamsungGemini Nanohttps://llmobile.kavents.com/ticker/artificial-analysis-mobile-benchmark/https://llmobile.kavents.com/ticker/artificial-analysis-mobile-benchmark/Mon, 24 Aug 2026 15:00:00 +0200 Chart: Artificial Analysis. Artificial Analysis has published a benchmark of small language models running locally on an iPhone 17 Pro. The study covers 41 quantised builds, 33 of which ran successfully on the device. Models were executed with llama.cpp at 4-bit quantisation or smaller, and had to fit within 8 GB of memory including the KV cache at 8K context. Artificial Analysis states that the inference benchmarking is operated in partnership with Liquid AI. Nanbeige4.2-3B and LFM2.5-2.6B tied for the highest intelligence score at 63. On speed and memory the two differ sharply: LFM2.5-2.6B completed the test prompt in 8.0 seconds using 2.3 GB, against 21.4 seconds and 4.0 GB for Nanbeige4.2-3B. Ornith-1.0-9B scored 62 and Qwen3.5 9B (Reasoning) 61. End-to-end generation time for a 1,024-token prompt plus 256 output tokens ranged from 0.9 seconds for LFM2.5-230M to 26.7 seconds for Falcon-H1R-7B. Peak in-test memory at 4K context ran from 0.4 GB to 6.9 GB. The intelligence index combines tool calling, instruction following, knowledge, scientific reasoning and mathematics. Artificial Analysis notes that context limits, not parameter count alone, held back several 9B models, which frequently hit a 16K ceiling. Chart: Artificial Analysis. Source: https://artificialanalysis.ai/articles/mobile-phone-intelligence-inference Read the article: https://llmobile.kavents.com/ticker/artificial-analysis-mobile-benchmark/BenchmarksiPhoneQuantisationllama.cppLiquid AIhttps://llmobile.kavents.com/ticker/ornith-1-5/https://llmobile.kavents.com/ticker/ornith-1-5/Wed, 19 Aug 2026 00:00:00 +0200Ornith has released Ornith-1.5, a model family with a 9B dense model, a 35B mixture-of-experts model that activates about 3B parameters per token and a 397B mixture-of-experts model. The 9B model also comes as Ornith-1.5-9B-Mobile, which the company says can be deployed on iPhone and Android devices. Ornith gives no size, memory or speed figures for the mobile build. The 9B model scores 47.0 on Terminal-Bench 2.1 with the Claude Code harness and 70.6 on SWE-bench Verified in Ornith’s tests. The company says the model matches or exceeds much larger models such as Gemma 4-31B and Qwen 3.6-35B. In the company’s chart, Qwen3.6-35B-A3B leads on SWE-bench Verified with 73.4 and on Terminal-Bench 2.1 with 52.5, while the 9B model scores 86.4 on GPQA Diamond and 54.2 on MCP-Atlas. The previous Ornith-1.0-9B reaches 43.1 on Terminal-Bench 2.1 in the same chart. Ornith's own benchmark figures for the 9B model. According to Ornith, its training loop lets the model propose new tasks, generate task-specific scaffolds and produce solution rollouts, with the reward from the rollouts propagated across all three stages. The company reports that the 35B model scores 67.8 on Terminal-Bench 2.1 with the Terminus-2 harness, against 52.5 for Qwen 3.6-35B. The models are on Hugging Face, with GGUF builds of all three sizes and MLX builds of the 9B and 35B models. Source: https://ornith.ai/ornith_1_5.html Read the article: https://llmobile.kavents.com/ticker/ornith-1-5/ModelsAppleAndroidhttps://llmobile.kavents.com/ticker/online-sdft/https://llmobile.kavents.com/ticker/online-sdft/Tue, 18 Aug 2026 16:57:46 +0200I-Ju Lin and Zhang-Wei Hong published Online-SDFT, a method that keeps fine-tuning a small language model on a phone from the user’s own stream of decisions. On a synthetic notification-routing benchmark with Liquid AI’s LFM2.5-230M, the authors report 70.28% of decisions matching a hidden user preference against 52.78% for the strongest baseline they tested, and cumulative regret of 44.76 against 105.26, over 720 decisions in three paired streams. They state that this comparison ran on Apple MPS rather than on a phone, and that the confidence intervals are nominal across only three streams. Self-distillation here means the model teaches itself. The model picks an action first, the phone later observes what happened, and the same model is then run again with that outcome added to its input. Its better-informed answer becomes the training target for the version that saw only the original context, so no second teacher model, no human label and no reward score is needed. The authors keep the 230M base frozen and train a LoRA adapter of 172,032 parameters across 48 tensors in six attention layers, with the teacher pass running with the adapter switched off so student updates cannot move the teacher. Feeding one interaction at a time into so small an adapter makes it noisy and prone to forgetting, so the authors keep a 64-row buffer of recent lessons and sample each batch from it. Their ablation puts that buffer ahead of the exploration schedule as the load-bearing part, with accuracy dropping from 70.28% to 39.58% and regret climbing from 44.76 to 134.90 once replay is removed, while dropping exploration instead leaves accuracy at 69.44% and regret at 62.00. Diagram: I-Ju Lin and Zhang-Wei Hong. The repository carries an Android proof of concept in which LFM2.5-230M decides and ONNX Runtime Training performs the adapter update on the device, with replay and checkpoints in app-private storage that survive a restart. Exporting the model still happens on a Linux host, and the deployment guide asks for Android 12 or newer on ARM64 with 12 GB of RAM at minimum, 16 GB preferred, and 8 GB of free storage, because the training graph is FP32. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: I-Ju Lin and Zhang-Wei Hong. Watch on the project page The authors call that graph a correctness implementation rather than a claim of practical background training on low-memory phones, and state that they did not measure Android latency, peak memory, battery use or thermal behaviour, and that the benchmark uses synthetic streams rather than real users. The code, the Android app and a Colab notebook are on GitHub under the MIT license. No paper or preprint accompanies the release. Source: https://lin826.github.io/SLM-Online-SDFT/ Read the article: https://llmobile.kavents.com/ticker/online-sdft/ResearchDistillationAndroidLFMOpen sourcehttps://llmobile.kavents.com/ticker/pixel-watch-5-offline-gemini/https://llmobile.kavents.com/ticker/pixel-watch-5-offline-gemini/Wed, 12 Aug 2026 19:30:00 +0200Gemini Intelligence is coming to the Pixel Watch 5, 9to5Google reports. According to the report, offline Gemini commands on the watch use a separate on-device model when the phone or an internet connection is unavailable. Those commands cover timers and alarms, brightness and modes, music control, opening apps and starting workouts. On-device smart replies now offer three responses instead of one, which the report attributes to a Gemini Nano upgrade that makes them 50 percent faster. Other parts of the feature set depend on a connection. Proactive Suggestions, formerly Magic Cue, are generated on a paired Pixel 11 and bridged to the watch, and Personal Intelligence draws on Gmail, Calendar and Keep. The update also brings a new At a Glance space on the watch face for timers, workouts, music, flight details and navigation. Source: https://9to5google.com/2026/08/12/pixel-watch-5-gemini-intelligence/ Read the article: https://llmobile.kavents.com/ticker/pixel-watch-5-offline-gemini/GooglePixelWearablesGemini Nanohttps://llmobile.kavents.com/ticker/pixel-11-tensor-g6/https://llmobile.kavents.com/ticker/pixel-11-tensor-g6/Wed, 12 Aug 2026 19:00:00 +0200Google has announced the Pixel 11, Pixel 11 Pro and Pixel 11 Pro XL, built around the Tensor G6 chip. Google states that Tensor G6 packs 50 percent more TPU compute and, paired with the latest Gemini Nano model, processes on-device AI tasks up to 3.5 times faster while using up to 3.5 times less energy. The company also cites an upgraded CPU with 25 percent faster web browsing and 15 percent quicker app launches, and says the chip powers the 30x Super Zoom on the 5x telephoto lens. Google does not publish RAM figures, model sizes or per-task latency in the announcement. Pre-orders opened on 12 August, with retail availability from 20 August. ▶Meet Google Pixel 11 ProLoading connects your browser to www.youtube-nocookie.com, which may process your IP address and use cookies. Load contentOpen externallyVideo: Made by Google. Source: https://blog.google/products-and-platforms/devices/pixel/google-pixel-11-pro-xl/ Read the article: https://llmobile.kavents.com/ticker/pixel-11-tensor-g6/GooglePixelChipsNPUGemini Nanohttps://llmobile.kavents.com/ticker/rikkahub-agent-local-llm/https://llmobile.kavents.com/ticker/rikkahub-agent-local-llm/Sat, 08 Aug 2026 00:00:00 +0200Adam Conway at XDA has tested RikkaHub Agent, a fork by ExTV of the open-source Android LLM client RikkaHub, which adds an agent layer with more than 80 device tools. The phone runs the agent loop and executes the tools; the model itself runs elsewhere on the local network over an OpenAI-compatible endpoint. The test device was an Oppo Find N5 with a Snapdragon 8 Elite and 16 GB of RAM. Inference ran on a Radeon RX 7900 XTX with Qwen 3.6 27B at Q4_K_M, roughly 16 GB, at 40 to 50 tokens per second. A second configuration used Qwen 3.6 35B on a Lenovo ThinkStation PGX with Nvidia’s GB10 Grace Blackwell. Screenshot: XDA. The tool set covers tapping, swiping, scrolling and typing on screen, reading notifications, sending SMS, file management, browser control, SSH, scheduled jobs, a JavaScript runtime and Telegram. All tools are off by default and enabled per assistant, actions with side effects require approval, and a deterministic blocklist rejects destructive commands. The default context is 7,000 input tokens, rising to 32,000 with every tool enabled. Asked whether whisper.cpp was installed, the agent determined that neither the command-line tool nor a model was present, requested permission, then installed the dependencies, cloned the repository and compiled it inside Termux. The build took seven minutes, and the agent chose the 75 MB small model on speed and accuracy grounds. It then transcribed a Telegram voice message, installing ffmpeg, converting the OGG/Opus file to 16 kHz mono WAV and returning a transcript in about two seconds. Running the model on the phone itself was slower. With Gemma 4 E2B, a 2.6 GB build accelerated by LiteRT-LM, Conway reports chat worked but agentic tasks were painful, and tool calls were emitted as plain text until an update fixed the behaviour. Frontier intelligence is never what I’d want from a phone agent, which is why a local LLM works perfectly. Source: https://www.xda-developers.com/turned-phone-ai-agent-powered-local-llm-installed-software/ Read the article: https://llmobile.kavents.com/ticker/rikkahub-agent-local-llm/AndroidAgentsQwenOpen sourceSpeechhttps://llmobile.kavents.com/ticker/lfm2-5-2-6b/https://llmobile.kavents.com/ticker/lfm2-5-2-6b/Tue, 04 Aug 2026 17:00:00 +0200Liquid AI has released LFM2.5-2.6B, a 2.6-billion-parameter model built on its LFM2 architecture for edge deployment. The company reports 30 tokens per second on a phone, with CPU inference staying under 2.5 GB of memory. Its published chart identifies the device as a Qualcomm Snapdragon Galaxy (SM-S948U1) and puts memory use at 2,448 MB at Q4_K_M quantisation with a 4K-token input context. Chart: Liquid AI. On other CPUs Liquid AI cites 220 tokens per second on an M5 Max and 113 tokens per second on a Ryzen AI Max+ 395. The model was pre-trained on roughly 34 trillion tokens, uses a 128K vocabulary, and its mid-training included a dedicated 128K context-extension phase. Liquid AI positions the model for agentic use, describing planning, tool calling and multi-step tasks, and states that it is competitive with, and often outperforms, models nearly four times its size on instruction-following and tool use, while larger models keep an advantage in coding. Weights are on Hugging Face in BF16, GGUF for llama.cpp, and ONNX. Source: https://www.liquid.ai/blog/lfm2-5-2-6b Read the article: https://llmobile.kavents.com/ticker/lfm2-5-2-6b/Liquid AIOpen weightsLFMBenchmarksAgentshttps://llmobile.kavents.com/ticker/gemini-nano-4-mlkit-prompt-api/https://llmobile.kavents.com/ticker/gemini-nano-4-mlkit-prompt-api/Wed, 22 Jul 2026 18:00:00 +0200Google’s Android developer blog states that Samsung’s new foldable devices come with Gemini Nano 4, which it calls its latest on-device model. The post credits Nano 4 with support for more than 140 languages and better multimodal understanding. Apps reach the model through ML Kit’s Prompt API, which sends natural language requests on-device to Gemini Nano. It takes text, or a combination of image and text, and returns text or structured output. Google names structured output and thinking mode as the features to use for on-device intelligence. The ML Kit release notes dated 14 July 2026 record the structured output API, system instructions and thinking mode arriving in the Prompt API, along with multi-image support and an output token limit raised to 4,096 tokens. A note dated 21 July records a fix for Gemini Nano v4 compatibility in the Prompt API on non-Pixel devices. The Prompt API moved from alpha to beta in January 2026 and carries no service level agreement or deprecation policy. The post also points developers to app functions, which share an app’s capabilities with the Gemini Intelligence system. Its remaining sections cover adaptive layouts, fold-aware design, CameraX and Wear OS widgets. Source: https://android-developers.googleblog.com/2026/07/optimize-galaxy-screen-sizes.html Read the article: https://llmobile.kavents.com/ticker/gemini-nano-4-mlkit-prompt-api/GoogleAndroidSamsungGemini NanoDeveloper toolshttps://llmobile.kavents.com/ticker/fblayout/https://llmobile.kavents.com/ticker/fblayout/Tue, 07 Jul 2026 12:18:25 +0200Researchers at the University of Macau, the University of Georgia and the Hong Kong University of Science and Technology published FBLayout on July 7, 2026, a system that fine-tunes transformer models on a phone’s own GPU, accepted at ACM MobiSys 2026. On a OnePlus Ace 5 Pro with a Snapdragon 8 Elite and its Adreno 830 GPU, the authors measure training on Llama 3.2 1B, Qwen2.5 1.5B and Gemma 2 2B at 3.9 to 4.1 times the speed of MNN, 4.3 to 4.9 times TFLite and 5.4 to 5.7 times TVM. Over the same three baselines they count 3.5 times fewer global memory reads and 4.2 times fewer cache misses. The authors time inference on the Snapdragon 8 Elite at roughly 6.5 times faster on the GPU than on the CPU, while fine-tuning models of 0.5B to 2B parameters on the GPU comes out level with the CPU or up to 43 percent worse. Their explanation is memory layout, meaning the order in which a tensor’s numbers physically sit in memory. Mobile GPUs hold tensors in a 2D texture grid whose cache rewards reading neighbouring cells, and training reuses forward-pass activations and weights in the backward pass, which sums along a different dimension and so walks the same tensor in strides instead of in rows. Each stride that misses the cache becomes a fetch from DRAM at under 100 GB/s of mobile memory bandwidth, and the GPU stalls waiting for it. Existing frameworks take one of two ways out, and the paper prices both. MNN reuses the forward layout in the backward pass, which the authors put at up to a 5.3 times slowdown on key operations and at under 21 percent arithmetic-unit utilisation in a matrix-multiply benchmark. TFLite instead inserts explicit transpose and reshape operators, and the paper puts that route at 40.8 to 50.0 percent of total runtime across the three LLMs and at roughly a third of all memory allocations. FBLayout defines one tile-based layout, called R-Tile, that keeps both directions contiguous, then rewrites the remaining reshapes as coordinate arithmetic so no data moves, which they report covers over 86 percent of all layout transformations and over 95 percent of those in the LLMs. The evaluation covers seven models, adding BERT-Large, ViT-Large, Whisper-Large and Stable Diffusion 1.5 to the three LLMs, with LoRA adapters in the attention and MLP blocks for everything except ViT. Beyond the Snapdragon 8 Elite phone the authors test a OnePlus Ace 10 Pro with a Snapdragon 8 Gen 1 and an Adreno 730 and a OnePlus Ace 5 Ultra with a Dimensity 9400+ and an Arm Mali Immortalis-G925, and report gains on both. Preparing the Llama 3.2 1B training graph takes 6.1 seconds in their measurement, against 723 seconds in MNN, 862 in TFLite and over 4,800 in TVM, and a Monsoon power monitor puts energy use over five training rounds at 3.5 to 6.3 times below the baselines. FBLayout is built on top of Alibaba’s MNN as about 9.2K lines of C++ and OpenCL, extending it with automatic differentiation so the training graphs exist in the first place. The paper names no code release and points to no repository. It appeared at MobiSys 2026 in Cambridge in June 2026 and is on arXiv under a CC BY-NC-ND 4.0 license. Source: https://arxiv.org/abs/2607.21624 Read the article: https://llmobile.kavents.com/ticker/fblayout/ResearchMemoryQualcommMediaTekAndroidhttps://llmobile.kavents.com/ticker/mlperf-inference-mobile/https://llmobile.kavents.com/ticker/mlperf-inference-mobile/Mon, 15 Jun 2026 17:00:00 +0200MLCommons announced MLPerf Mobile v6.0 on June 15, 2026, adding generative AI tests that run Llama on an Android device and report token throughput. The new workload comes in 1B, 3B and 8B parameter sizes, using Llama 3.2 1B Instruct, Llama 3.2 3B Instruct and Llama 3.1 8B Instruct. Each size runs twice, once against TinyMMLU, a small multiple-choice knowledge set, and once against IFEval, which checks whether a model obeys the instructions in a prompt, according to the task configuration in the app repository. MLCommons says the tests run on devices with enough memory through the CPU without tailored acceleration, and that v6.0 adds NPU-accelerated execution of Llama 3.1 8B Instruct on Qualcomm Snapdragon 8 Elite Gen 5. The v6.0 app release of June 9, 2026 also brings support for MediaTek Dimensity 9500 chips and updates support for the Samsung Exynos 2600. The rest of the suite covers image classification with MobileNetV4 on ImageNet, object detection on COCO 2017, segmentation with MOSAIC on ADE20K, language understanding with MobileBERT on SQuAD 1.1, super resolution with EDSR, and Stable Diffusion 1.5 on COCO captions, as the benchmark page lists them. Almost every test runs in the single stream scenario, where the score is the 90th percentile latency for one query at a time, and image classification also runs an offline pass scored as throughput. Stable Diffusion, the first generative test in the suite, arrived in the v4.1 app release of February 11, 2025. Mostafa El-Khamy, co-chair of the MLPerf Mobile working group, had set out the plan in May 2024. Looking ahead, the MLPerf Mobile group is working to add generative AI tasks to a future version of the benchmark suite. Results come from chip and device makers that run the app and submit their logs. The published v6.0 results hold one submission in the closed division, from Samsung on an Exynos 2600 phone, and it covers the vision tests, language understanding, super resolution and Stable Diffusion rather than the new Llama ones. Samsung’s logs give a 90th percentile single stream latency of 0.56 ms for image classification, 0.83 ms for MobileBERT and 1.87 s for one Stable Diffusion image. The v5.0 round carried submissions from Samsung and Qualcomm Innovation Center. Screenshots: Samsung's submission to MLPerf Mobile v6.0, published by MLCommons. Submissions are held back until review closes under the MLCommons submission rules, which keep every result confidential inside a review committee after the deadline and let submitters file objections that the committee then decides. MLCommons publishes the accepted runs in the results table on the MLPerf Inference Mobile page, with the logs and calibration notes in a per-round repository on GitHub. The app is available through Google Play, the Apple App Store and the MLPerf Mobile repository under the Apache 2.0 license. Source: https://mlcommons.org/2026/06/mlperf-mobile-v6/ Read the article: https://llmobile.kavents.com/ticker/mlperf-inference-mobile/BenchmarksAndroidLlamaNPUQualcommhttps://llmobile.kavents.com/ticker/on-device-diffusion-llm-npu/https://llmobile.kavents.com/ticker/on-device-diffusion-llm-npu/Thu, 11 Jun 2026 14:44:00 +0200Tuowei Wang, Yanfan Sun and Ju Ren of Tsinghua University and Beihang University published llada.cpp on June 11, 2026, an inference framework that runs diffusion language models on a phone’s NPU. On an OnePlus Ace5 Pro with a Snapdragon 8 Elite, the authors measure LLaDA-8B-Instruct generating 128 tokens 38 times faster than the same model on the phone’s CPU for GSM8K prompts and 42 times faster on BoolQ, and they report a range of 17 times to 42 times across their phones and tasks. The same figure puts Llama-3-8B-Instruct on the NPU at 19 to 21 times that CPU baseline, and the authors state that llada.cpp runs LLaDA-8B up to 3.9 times faster than an autoregressive model of the same size. Chart: Wang, Sun and Ren, CC BY 4.0. A diffusion language model does not emit one token after another. It starts from a block of masked positions, 32 tokens by default here, and refines all of them together over repeated denoising steps, committing the positions it is most confident about and carrying the rest into the next step. That turns decoding into a run of large matrix operations, which is the work a mobile NPU is built for, while autoregressive decoding hands the NPU one token’s worth of work per step. The cost the paper identifies is that every denoising step recomputes the full sequence, so the parallel work gets repeated many times over. llada.cpp is built on llama.cpp and targets Qualcomm’s Hexagon NPU with 4-bit weights. Once a block is nearly done only a few masked tokens are left, so the framework pulls tokens from the next block into the same NPU pass to keep it full, keeps already committed tokens open to revision and repairs the unstable ones on the CPU while the NPU carries on, and repacks the limited address space the NPU can see so weights and cache do not have to be remapped between steps. The authors’ breakdown of a 128-token GSM8K request on the Snapdragon 8 Elite puts it at 2996.2 s on the CPU, 607.0 s once the prefix KV cache is reused, and 16.1 s with the full framework. Accuracy moves in both directions. Measured on 200-sample subsets against LLaDA-8B-Instruct on the CPU, the authors’ table has GSM8K rising from 39.0 to 43.5 and ARC-C from 84.0 to 85.0, while BoolQ falls from 82.5 to 80.5 and HellaSwag from 51.0 to 49.5, and their ablation shows the borrowed future-block tokens costing accuracy on all four tasks before the CPU-side revision path recovers it. The work covers three 16 GB phones, an OnePlus 12 with a Snapdragon 8 Gen 3, the Ace5 Pro, and an OnePlus 15 with a Snapdragon 8 Elite Gen 5, and the authors state that on the Snapdragon 8 Gen 3 their framework is still slower than Llama-3-8B at 128 tokens and only pulls ahead on the two newer chips. The paper is on arXiv under CC BY 4.0 and names no code release. Source: https://arxiv.org/abs/2606.13740 Read the article: https://llmobile.kavents.com/ticker/on-device-diffusion-llm-npu/QualcommNPUAndroidllama.cppResearchhttps://llmobile.kavents.com/ticker/caped-privacy-defence/https://llmobile.kavents.com/ticker/caped-privacy-defence/Wed, 10 Jun 2026 22:48:00 +0200Researchers at the Chinese University of Hong Kong, Southeast University and Shandong University published CAPED on June 10, 2026, a filter that blacks out parts of a phone’s screenshots before they reach a cloud-hosted GUI agent. Across a 28-task privacy suite the authors built themselves, weighted seeded leakage falls from 0.766 with raw screenshots to 0.268 with the filter in front, counted only on runs where the agent still finished its task. A multimodal model plays the attacker and scores what it can still infer from the whole screenshot sequence, not from single frames. A screenshot-based agent operates a phone the way a person does, by looking at the screen, so every image it uploads carries whatever happens to sit next to the thing the user asked for. The authors call this incidental visual privacy exposure and name contacts, message previews, non-target photos, filenames, recommendation feeds and health widgets as content the remote model receives although the user never mentioned it. In one of their test shops, a request to buy sunglasses puts prenatal vitamins, a baby bottle set and a maternity dress in front of the model, because the shop’s recommendation rows share the page with the product. Stripping names and numbers out of the text does not close that gap, and the authors measure text-only anonymisation at 0.752 against 0.766 for no protection at all. Figure: Shen et al., licensed CC BY 4.0. The mock SunShop task asks the agent to buy sunglasses, so the personalised rows around it count as incidental exposure. CAPED decides per interface element rather than per screen, according to the paper. It first sorts the current screen into one of three postures, so galleries, maps, files and financial views show their text but hide their images, while messaging, contacts, feeds and health dashboards hide both by default. The user’s instruction is turned into a short list of what the agent is allowed to need, and a hidden element is released only when it matches that list, checked either by text matching or by asking a vision model whether a cropped thumbnail really shows the requested subject. Hidden areas keep their place in the layout under a solid overlay with an eye icon, and names in exposed text are replaced by placeholders the remote agent can still follow. The defence is meant to run on the phone, and the authors argue the instruction has to be interpreted locally too, since it can name people, relationships, locations and intentions before any screen policy applies. The prototype they measured does not run on a phone. They state that CAPED’s components ran on the evaluation host with an evaluation-side service for the heavier models, and that the timings should be read as live-loop prototype latency rather than physical-phone latency, energy or thermal behaviour. Over 1,394 measured steps the privacy work added 2.325 seconds per agent step, 22.6 percent of a 10.835-second step, most of it the detector at 1.811 seconds, plus 5.716 seconds once per task to extract the requirements. The protection also costs task success. On AndroidWorld’s 116 ordinary tasks, GUI-Owl-7B completed 77 of them (66.4%) unprotected and 64 (55.2%) behind the full filter, which the authors call a prototype-level utility cost. They name further limits themselves, writing that screens their rule base does not cover fall back to the most permissive posture, which they describe as not privacy-conservative, and that open-ended requests such as finding the latest receipt widen exposure and account for much of the leakage that remains. On the connected Gmail, Drive and Photos tasks the filter moves leakage from 0.402 to 0.347 only, the interactive unmasking path was never tested with real users, and the authors write that they plan to release the CAPED artifact in a future public version. Source: https://arxiv.org/abs/2606.12666 Read the article: https://llmobile.kavents.com/ticker/caped-privacy-defence/AgentsAndroidPrivacyResearchBenchmarkshttps://llmobile.kavents.com/ticker/apertus-mini/https://llmobile.kavents.com/ticker/apertus-mini/Wed, 27 May 2026 23:40:00 +0200Researchers working in the Swiss AI Initiative published a technical report on May 27, 2026 describing three language models distilled from Apertus 8B, at 0.5B, 1.5B and 4B parameters. The authors are based at EPFL, ETH Zurich and ISTA, and the models sit on Hugging Face as the Apertus Mini collection. They inherit what the project calls a fully open and compliant release, meaning published weights, published data pipeline and published recipe, with pre-training restricted to openly available data that honours robots.txt opt-outs. The model card lists 1,811 natively supported languages. Each instruction-tuned model also ships in MLX format at 3, 4 and 6 bits, which the authors aimed at Apple devices and at memory-limited mobile and edge deployment. The 4-bit 4B weights are a 2.59 GB file against 7.65 GB for the same model in bfloat16, and the 1.5B and 0.5B come to 1.08 GB and 0.31 GB. The authors build the quantised copies with quantisation-aware distillation, a short recovery run that retrains the compressed weights against the 8B model’s own output distribution, and report 90% to 104% of the unquantised few-shot accuracy on ARC, HellaSwag, MMLU and WinoGrande. Separate FP8 and NVFP4 checkpoints target NVIDIA GPUs instead. The base model table averages ARC, HellaSwag, WinoGrande, XNLI, XCOPA and PIQA, and puts the three sizes at 51.79, 56.66 and 61.53 against 64.96 for the 8B teacher. The same table gives the Qwen3 base models 52.23, 57.51 and 62.14 at comparable sizes, so the distilled models land just under them on the average while leading on the two cross-lingual sets, 45.03 against 43.00 on XNLI and 63.82 against 61.82 on XCOPA at 4B. On a multilingual average of MMLU, TruthfulQA, ARC, instruction following and LogiQA for the instruction-tuned models, the authors report 0.318, 0.382 and 0.473 against 0.401, 0.457 and 0.521 for Qwen3 and 0.497 for Gemma 3 4B, and write that the family is weaker at instruction following and maths. Each model saw 1.7T tokens taken from the final and most heavily filtered phase of the original Apertus data, with no new sources added, packed into 4,096-token sequences and trained on 64 GH200 GPUs at the Swiss National Supercomputing Centre. The cost table puts the 4B run at 2.0E22 FLOPs and the single pass that produced the teacher’s logits for all three students at 1.4E22, against 3.7E23 FLOPs for the original Apertus 8B pre-training on 15T tokens, which the authors describe as making the whole family cost under 12% of the teacher. Swiss AI released 24 checkpoints in total, base, instruction-tuned and quantised, under the Apache 2.0 licence with an acceptable use policy attached, along with the training, post-training and quantisation code. Source: https://arxiv.org/abs/2605.29128 Read the article: https://llmobile.kavents.com/ticker/apertus-mini/Swiss AIDistillationQuantisationOpen weightsApple Siliconhttps://llmobile.kavents.com/ticker/mobilemoe/https://llmobile.kavents.com/ticker/mobilemoe/Tue, 26 May 2026 19:58:00 +0200Meta AI published MobileMoE on May 26, 2026, a family of three language models for phones that hold 1.3B, 2.8B and 5.3B parameters but run only 272M, 528M and 922M of them on any given token. Each one is a sparse mixture of experts, meaning the feedforward block in every layer is split into 60 small networks plus one always-on shared network, and a router picks 4 of the 60 for each token while the rest sit idle. Per-token compute therefore follows the active count, and the memory footprint follows the total. That split is what makes the models awkward to host on a phone. The authors quantise all three to 4 bits with quantisation-aware training, which puts the weights at 0.68 GB, 1.48 GB and 2.75 GB, because every expert has to be reachable even though only a few run per token. On a Samsung Galaxy S25 they measure peak resident memory during inference at 1.49 GB for the smallest model at an 8k context against 1.91 GB for the dense MobileLLM-Pro, and at 4.71 GB for the largest. Sizing the whole expert set to fit phone DRAM is the opposite of the route Edge0 takes, which streams expert weights off storage to keep a 35B model inside a couple of gigabytes. The paper reports that peak memory under real prompts runs 1.2 to 2.1 times higher than under repeated dummy tokens for the mixture-of-experts models, while staying flat for the dense baseline, because varied input routes to more experts and pulls more of them into RAM. Meta AI therefore profiled with real code, knowledge and maths prompts and calls dummy-prompt figures a lower bound rather than a measurement. The authors ran the models through ExecuTorch on the Galaxy S25 with a Snapdragon 8 Elite on 4 CPU threads and on an iPhone 16 Pro with an A18 Pro on 2 CPU threads, after writing a fused mixture-of-experts operator because mobile CPU backends ship none. On a 512-token prompt they measure the smallest model decoding at 130.0 tok/s on the Galaxy S25 against 56.5 for MobileLLM-Pro, and at 180.5 tok/s on the iPhone 16 Pro against 55.1. Time to first token on a 1k prompt drops from 4.26 s to 2.01 s on the Galaxy and from 6.03 s to 2.14 s on the iPhone. Across both phones, CPU and GPU, and the XNNPACK and MLX backends, Meta AI reports the smallest model running 1.8 to 3.8 times faster at prefill and 2.2 to 3.4 times faster at decode than the dense baseline at comparable 4-bit weight memory. Across 14 benchmarks the authors score the instruction-tuned models at 46.7, 55.3 and 60.1 on average at 16-bit precision, against 42.7 for the dense Qwen3.5 0.8B, 44.0 for Gemma 3 1B, 46.9 for Llama 3.2 1B and 50.8 for Qwen3.5 2B, the last of which carries 1.9B active parameters. The largest model reaches 60.1 against 55.6 for Ai2’s OLMoE-1B-7B, with 30 percent fewer active and 23 percent fewer total parameters. After 4-bit quantisation those averages fall to 44.0, 52.5 and 57.8, which leaves the smallest model 1.5 points behind MobileLLM-Pro at 45.5 while scoring 9.4 points higher on MMLU. Meta AI states that the training data across all four stages comes from openly licensed public datasets, names no weight release, and lists mobile NPU deployment, expert pruning and mixed-precision quantisation as further work. Chart: Meta AI. The authors' own comparison of instruction-tuned models at 16-bit precision, with the horizontal axis giving total parameters alongside the 4-bit weight footprint those imply. Source: https://arxiv.org/abs/2605.27358 Read the article: https://llmobile.kavents.com/ticker/mobilemoe/MetaMixture of expertsMemoryQuantisationiPhonehttps://llmobile.kavents.com/ticker/litert-lm-on-device-genai/https://llmobile.kavents.com/ticker/litert-lm-on-device-genai/Tue, 19 May 2026 18:00:00 +0200Google has detailed LiteRT-LM, the runtime it uses to run Gemma 4 on-device in Chrome, ChromeOS, the Pixel Watch and the Google AI Edge Gallery app. It sits on top of LiteRT, formerly TensorFlow Lite, and adds GenAI libraries over the XNNPACK and ML Drift kernels. Running Gemma 4 E2B without multi-token prediction, Google measures 52 decode tokens per second on the Android GPU through OpenCL, 56 on iOS through Metal, and up to 76 in the browser on WebGPU. The Android figures come from a Samsung S26 Ultra, iOS from an iPhone 17 Pro, and web from Chrome on a 2024 MacBook Pro with an M4 Max. Chart: Google. The published comparison puts LiteRT-LM prefill at 3,808 tokens per second against 292 for llama.cpp on the Android GPU, 2,878 against 1,493 for MLX on iOS, and 4,182 against 1,127 for ONNX on the web. Decode figures in the same chart are 52 against 19, 56 against 32 and 76 against 29. Multi-token prediction raises decode further. Google reports up to a 2.2x speedup, taking Gemma 4 E2B from 52 to 85 tokens per second and E4B from 21 to 47, measured on a Samsung S26 Ultra GPU. The drafter and the primary model run on the same hardware block so the shared KV cache and activations stay in local memory. Chart: Google. On memory, LiteRT-LM keeps per-layer embeddings out of memory and loads image and audio encoders only when a task needs them. Google states that the 2.58 GB Gemma 4 E2B model runs with a physical footprint of 607 MB on Apple mobile CPUs using XNNPACK weight caching. For agentic use the runtime supports Thinking Mode, which developers can stream to the interface or strip to save KV cache space: By dedicating a scratchpad for step-by-step reasoning before the model commits to an action, LiteRT-LM can significantly improve the output quality. Constrained decoding enforces JSON schemas on tool payloads, and native function calling pauses execution, returns the tool-call request to the app and resumes on the tool’s output. Session save and restore serialise KV cache state so long conversations skip the prefill phase on return. Beyond the existing Kotlin and C++ interfaces on Android, Google is adding an open-source Swift API for Apple platforms and a JavaScript API for the web through WebAssembly and WebGPU. Source: https://developers.googleblog.com/blazing-fast-on-device-genai-with-litert-lm/ Read the article: https://llmobile.kavents.com/ticker/litert-lm-on-device-genai/GoogleLiteRTGemmaBenchmarksAndroidhttps://llmobile.kavents.com/ticker/quant-npu/https://llmobile.kavents.com/ticker/quant-npu/Tue, 19 May 2026 12:48:00 +0200Eight authors published Quant.npu on May 19, 2026, a way of quantising 1B to 3B language models so they run on a phone’s neural processing unit with no rescaling at runtime. On a Qualcomm SM8650 NPU they measure their 4-bit weight, 8-bit activation build of Llama-3.2-3B-Instruct finishing up to 15.1 percent faster than the same model under ExecuTorch’s 4-bit per-block weights with 16-bit activations, for about 1 percentage point of zero-shot accuracy. Across every model and task in the paper the authors put the average cost at a 2.58 percent accuracy drop and a 1.23 rise in perplexity. Quantisation stores a tensor as small integers plus a scale factor that maps them back to real values. Most published 4-bit methods, including the QuaRot and SpinQuant work Quant.npu is measured against, work that scale out while the model runs by scanning each activation tensor for its largest value. A mobile NPU compiles its graph before the model ships and expects every scale to be fixed by then, because that scan is a reduction the hardware is poor at, at least 7 vector instruction cycles for a 128-element 8-bit vector by the authors’ count, and because two of the four rotation steps these methods use against outliers need floating-point matrix multiplications during inference, which the paper puts at roughly double the compute. Anything left dynamic therefore has to run outside the compiled integer pipeline, and the authors report that simply freezing a model optimised under dynamic quantisation costs 15.61 percent accuracy on SmolLM2-1.7B-Instruct. Quant.npu learns the scales during optimisation instead and folds its two rotation matrices into the weights offline, so the shipped graph is integer arithmetic end to end. In the paper’s main table for Llama-3.2-3B-Instruct at 4-bit weights and 8-bit activations, that lifts average zero-shot accuracy across six benchmarks to 58.27 percent against 46.23 for SpinQuant and 47.66 for QuaRot in the same static setting, with perplexity on C4 falling from 28.78 to 19.16. Against the slower ExecuTorch configuration it beats on latency, though, Quant.npu is the one behind, scoring lower on five of the six benchmarks, 1.08 points lower on average and 0.97 higher in perplexity, and FP32 stays ahead of both at 61.3 percent. On the newer SM8750 NPU the authors report peak memory of 1876 MB to 1985 MB for their 3B builds against 2216 MB to 2435 MB for ExecuTorch, and prefill of 969.72 tok/s for Qwen2.5-3B-Instruct on the DroidTask set against 808.10. They name two limits of their own, that the pipeline still keeps a few operations such as the SiLU activation at 16 bits to hold accuracy, which they call a bottleneck against a fully low-bit pipeline, and that they use no method for picking calibration data even though the optimisation is sensitive to it. Quantising Qwen3-1.7B takes about 2.5 hours on NVIDIA A40 GPUs, and the preprint announces no code release. Source: https://arxiv.org/abs/2605.20295 Read the article: https://llmobile.kavents.com/ticker/quant-npu/QualcommNPUQuantisationLlamaResearchhttps://llmobile.kavents.com/ticker/hunyuan-hy-mt-1-5-offline-translation/https://llmobile.kavents.com/ticker/hunyuan-hy-mt-1-5-offline-translation/Wed, 29 Apr 2026 15:55:00 +0200Tencent Hunyuan has open-sourced Hy-MT1.5-1.8B-1.25bit, a translation model that runs fully offline on a phone. Quantisation to 1.25 bits cuts the file from 3.3 GB at FP16 to 440 MB; a 2-bit build is 574 MB. The announcement states the result is 25 percent smaller and around 10 percent faster than prior 1.67-bit approaches with no loss of accuracy. Chart: Tencent Hunyuan. The compression uses Sherry, a ternary quantisation framework with 3:4 fine-grained sparsity: of every four weights, the three most important are stored in 1 bit as -1 or +1 and the fourth is zeroed. A custom STQ kernel aligns the format to the SIMD instruction set of mobile CPUs. Running it requires llama.cpp with STQ1_0 kernel support, added in pull request 22836, and the model card links a side-by-side speed demo recorded on a Snapdragon 888 with 8 GB of RAM. The model covers 33 languages, 5 dialects and minority languages, and 1,056 translation directions, including Tibetan and Mongolian. On the FLORES-200 benchmark Tencent reports it ahead of Tower-Plus-72B, Qwen3-32B, Microsoft Translator and Doubao on Chinese-to-foreign pairs, and the announcement claims it outperforms Google Translate and matches 235B-scale models. Chart: Tencent Hunyuan. Weights are on Hugging Face alongside an Android demo APK for offline translation. The code is released under the AngelSlim licence rather than a standard open-source licence. Source: https://x.com/TencentHunyuan/status/2049487799850840334 Read the article: https://llmobile.kavents.com/ticker/hunyuan-hy-mt-1-5-offline-translation/TencentOpen weightsQuantisationllama.cppTranslationhttps://llmobile.kavents.com/ticker/airgap/https://llmobile.kavents.com/ticker/airgap/Wed, 29 Apr 2026 08:06:11 +0200Xavier Puspus published Airgap on April 29, 2026, a React Native starter kit for branded customer support apps that have to keep answering when the network is slow, missing or deliberately blocked. Its configuration file pins a Gemma 4 E2B instruction-tuned model in Q3_K_S GGUF form, about 2.4 GB, which the app downloads on first launch and runs through llama.rn, the React Native binding for llama.cpp. The README states that Airgap searches bundled local documents first and uses a model only to phrase what that retrieval returned, and it lists Apple Foundation Models and Google’s ML Kit GenAI Prompt API as two further on-device answer providers an operator can put ahead of the downloaded file. The repository ships seven example configurations, for airline, banking, electric utility, healthcare, insurance, telecom and water utility support, each with its own branding, prompts and JSON knowledge files, and the downloaded-model path targets iOS 15.1 and newer and Android API 24 and newer. Airgap is published under the MIT license. It is an early single-author project, with 54 commits and no tagged release, and almost all of the code arrived in one squashed initial commit followed by a burst of work in August 2026. The README lists physical-device evaluation of the Apple and Android system providers as an open release gate and labels the published demo recordings as Android emulator and iOS simulator footage rather than evidence of either system model running on a phone. Source: https://github.com/xmpuspus/airgap Read the article: https://llmobile.kavents.com/ticker/airgap/Open sourceAndroidiOSGemmallama.cpphttps://llmobile.kavents.com/ticker/gemma-4-edge-agent-skills/https://llmobile.kavents.com/ticker/gemma-4-edge-agent-skills/Thu, 02 Apr 2026 18:00:00 +0200Google DeepMind has launched Gemma 4 under the Apache 2.0 licence, and the Google AI Edge team has described how the E2B and E4B variants run on edge hardware. The models support more than 140 languages and a 128K context window. Gemma 4 enables multi-step planning, autonomous action, offline code generation, and even audio-visual processing, all without specialized fine-tuning. Image: Google. The Google AI Edge Gallery app on iOS and Android gains Agent Skills, which Google calls one of the first applications to run multi-step, autonomous agentic workflows entirely on-device. The examples given are querying Wikipedia, turning speech input into charts and flashcards, pairing photos with generated music, and building a small app that plays animal calls through conversation alone. For in-app deployment Google points to LiteRT-LM. It reports that Gemma 4 E2B runs in under 1.5 GB of memory on some devices using 2-bit and 4-bit weights with memory-mapped per-layer embeddings, and that GPU optimisations process 4,000 input tokens across two skills in under 3 seconds. Constrained decoding enforces structured output, and dynamic context lengths let a single model span CPUs and GPUs. On smaller hardware, LiteRT-LM reaches 133 prefill and 7.6 decode tokens per second on a Raspberry Pi 5 CPU, and 3,700 prefill and 31 decode tokens per second on the NPU of a Qualcomm Dragonwing IQ8, the processor behind the Arduino VENTUNO Q. Gemma 4 is available with CPU and GPU support on Android and iOS, system-wide on Android through the new AICore Developer Preview, on Windows, Linux and macOS via Metal, and in the browser through WebGPU. Google also released a Python package and a litert-lm CLI for Linux, macOS and Raspberry Pi, with tool calling included. Source: https://developers.googleblog.com/bring-state-of-the-art-agentic-skills-to-the-edge-with-gemma-4/ Read the article: https://llmobile.kavents.com/ticker/gemma-4-edge-agent-skills/GoogleGemmaAgentsOpen weightsLiteRThttps://llmobile.kavents.com/ticker/llm-inference-at-the-edge/https://llmobile.kavents.com/ticker/llm-inference-at-the-edge/Tue, 24 Mar 2026 19:28:00 +0100Four authors at Conscious Engines published a benchmark of Qwen 2.5 1.5B at 4-bit across four edge platforms on March 24, 2026, feeding each device the same 258-token prompt 20 times in a row with a 1-second gap instead of once. On an iPhone 16 Pro the first two runs peak at 40.49 tokens per second, and from run 17 onwards the phone holds a plateau of 23.67 tok/s, which the authors put at 41.5 percent below that peak. The iPhone passes through the three thermal states iOS reports to an app. Runs 1 and 2 sit in Normal and average 36.83 tok/s, runs 3 to 16 sit in Warm and average 24.96 tok/s, and runs 17 to 20 sit in Hot at 23.67 tok/s with a variation of 1.2 percent. The authors write that the 1-second gap allows no thermal recovery, and they measure 5 percent of battery drained over the 20 runs, which they project to roughly 400 replies of this length per charge. A Samsung Galaxy S24 Ultra degrades more gently in the same test. Its Adreno 750 GPU boosts to 1,000 MHz on runs 4 and 5 for a peak of 12.21 tok/s, then steps down to a 720 to 770 MHz plateau from run 8 where the authors measure 10.38 tok/s, 15 percent off the peak, with a low of 9.55 tok/s on the final run. They stress that reaching this plateau took deliberate tuning, with the display off, the context capped at 2,048 tokens and MLC-LLM’s prefill chunk cut to 128 tokens, and that earlier unconstrained runs hit an OS-enforced GPU frequency floor. Neither phone ran the model on its NPU. The paper states that MLX dispatches to the A18 Pro GPU through Metal and does not target Apple’s Neural Engine, and that the S24 Ultra ran through MLC-LLM on the Adreno GPU, leaving the Snapdragon 8 Gen 3’s Hexagon NPU outside the measurement. The one NPU that did run the model, a Hailo-10H module in a Raspberry Pi 5, was also the only part that never throttled, holding 6.914 tok/s across all 20 runs with a variation of 0.04 percent at under 2 W, its CPU at 52.7°C and the NPU at 58.5°C, with attention layers that hailo-ollama does not offload falling back to the Pi’s Cortex-A76 CPU. A user running llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone reported 12.5 tok/s of generation on Gemma 3 4B and said the phone stayed cool. A laptop RTX 4050 on battery led on speed at 131.7 tok/s and 34.12 W, its GPU climbing from 55°C to 70°C without throttling. Energy per token comes out at 270.5 mJ for the Hailo module measured across the whole system, 297.3 mJ for the RTX 4050 measured at the GPU, and 143.0 mJ for the S24 Ultra plateau derived from the Android fuel gauge with the display off. The authors state that the three readings differ in scope and call the resemblance suggestive rather than settled, and report no energy figure for the iPhone because iOS exposes no per-component power reading. They tested one model, one prompt and one unit per platform across four different runtimes, and describe the results as characterising hardware and software together rather than hardware on its own. Source: https://arxiv.org/abs/2603.23640 Read the article: https://llmobile.kavents.com/ticker/llm-inference-at-the-edge/BenchmarksNPUiPhoneSamsungResearchhttps://llmobile.kavents.com/ticker/mobilellm-flash/https://llmobile.kavents.com/ticker/mobilellm-flash/Mon, 16 Mar 2026 23:10:00 +0100Meta AI published MobileLLM-Flash on March 16, 2026, three models of 350M, 650M and 1.4B parameters whose shapes were picked by measuring how fast each candidate actually ran on a phone. Meta reports the family reaching up to 1.8 times faster prefill and 1.6 times faster decode than Liquid AI’s LFM2 models on mobile CPUs, at comparable or higher accuracy, with support for up to 8k tokens of context. Meta’s first MobileLLM paper argued in 2024 that deep and thin networks win below a billion parameters, and the Flash paper states that such deep-and-thin structures often fail to improve on-device latency. Meta exported 100 architectures to a phone and found parameter count a poor predictor of measured prefill and decode speed, at a Kendall rank correlation of 0.40, with FLOPs little better at 0.46 and 0.55. The 1.4B model therefore runs 16 layers at a width of 2048, against the 30 layers at 1280 of MobileLLM-Pro-1B. Meta ran the latency tests on a Samsung Galaxy S25, a phone with a Snapdragon 8 Elite chipset, an octa-core CPU and 12 GB of memory, exporting each model through ExecuTorch to the XNNPACK CPU backend at 4-bit weights, 8-bit dynamic activations and a quantised KV cache, on 4 CPU threads and averaged over three runs after a warmup. At a 2k context the 650M model reaches its first token in 3.34 s against 6.01 s for LFM2 700M, and decodes at 85.35 tok/s against 53.57 tok/s. Meta repeated the 1k-context test on an iPhone 17, where the 650M model takes 2.81 s to the first token against 3.40 s for LFM2 700M. Most of the speed comes from letting some layers skip attention, so a token passes through those blocks without being compared to every earlier token. The 350M model keeps full attention in 7 of its 12 layers and the 650M model in 8 of 13, while the 1.4B model attends in all 16. Meta reports that skipping consistently beat sliding window attention, which instead limits each token to a fixed span of recent ones, and that more than three skipped layers in a row costs accuracy at the same latency, dropping TriviaQA exact match from 33.2% to 8.8%. Meta built the three models by pruning a shallow 1.8B variant of MobileLLM-Pro and inheriting its weights, which the paper says took about 35% of the tokens that pretraining MobileLLM-Pro required. Across nine reasoning and knowledge tasks Meta reports averages of 45.46, 48.57 and 55.06 for the three sizes, against 44.92, 48.52 and 50.48 for LFM2 at 350M, 700M and 1.2B, 51.79 for Gemma 3 1B and 49.01 for Llama 3.2 1B. After instruction tuning, the 1.4B model scores 47.89 on MMLU and 46.34 on HumanEval, against 49.30 and 37.80 for Llama 3.2 1B. The paper announces no release of the weights. Chart: Meta. Bubble area shows parameter count, latency measured on a Samsung Galaxy S25. Source: https://arxiv.org/abs/2603.15954 Read the article: https://llmobile.kavents.com/ticker/mobilellm-flash/MetaResearchBenchmarksAndroidPyTorchhttps://llmobile.kavents.com/ticker/flexserve/https://llmobile.kavents.com/ticker/flexserve/Tue, 10 Mar 2026 01:31:25 +0100Researchers at Shanghai Jiao Tong University published FlexServe on March 10, 2026, a serving system that keeps an on-device language model’s weights and the text going through it inside ARM TrustZone’s secure world. Running Llama 3.2 3B, Llama 3.1 8B, Qwen3 1.7B and Qwen3 8B quantised to 8-bit on a NanoPC-T6 board with a Rockchip RK3588, 16 GB of memory and a 6 TOPS NPU, the authors measure time to first token 4.41 percent above the same inference run without any protection, averaged over their prompt lengths. Against a TrustZone baseline that already uses a secure NPU and a loading pipeline, they report FlexServe 1.43 to 2.42 times faster, 1.85 times on average. TrustZone splits the chip into a normal world, where Android and every app runs, and a secure world the normal world cannot read. FlexServe puts the inference engine, the weights, the prompt, the generated text and the KV cache on the secure side, so a kernel an attacker has taken over cannot read any of it out of memory, and the authors justify that assumption with Linux’s 40 million lines of code and 9756 CVEs. Both halves are worth stealing in their account, since a vendor’s fine-tuned weights cost real money to train yet sit on a stranger’s device, and the apps calling a shared model feed it chat history and screen contents. The authors name the boundary they cannot cross, which is that an app in the normal world still receives its own input and output there, so a compromised kernel can take those, and only the weights stay protected in that case. Plain TrustZone is slow at this because it hands out secure memory as a few large contiguous regions. Reserving 8 GB at runtime therefore means the kernel’s contiguous allocator compacting scattered pages, which the authors time at 6440.67 ms against 568.58 ms for FlexServe’s page-granular secure memory, and the secure world cannot normally touch the NPU at all, leaving 30.06 s of CPU computation where the NPU needs 1.94 s. FlexServe carves secure pages out with the stage-2 page tables that ARM’s virtualisation extension already provides, and flips the NPU between protected and unprotected mode in 0.21 ms. Handing 8 GB back drops from 732.85 ms to 80.50 ms, because the pages are released lazily rather than zeroed in one pass. A scheduler decides which models stay resident when several share the device, holding the earliest layers of each in secure memory so the pipeline does not stall and prefetching the next model while the current one is still generating. With a 4 GB cache and requests drawn at random from UltraChat, OpenAssistant, Dolly and Alpaca, the authors measure pairs of models answering 8.84 times faster than the plain secure-world baseline and 2.53 times faster than the optimised one. Across ten agent workflows, where one model’s finished answer feeds the next, the gains reach 24.30 times and 4.05 times, averaging 14.15 and 2.94. Serving one phone-side model to several callers is also what ElastiLM works on, by resizing the shared model to each request’s latency budget rather than isolating it. The protection costs the rest of the device something as well. A SQLite instance holding 3 GB in the normal world runs 1.31 times slower while FlexServe holds 8 GB in the background, against 3.27 times slower with the contiguous allocator, and the authors put the standing virtualisation cost at 2.46 percent on SPEC CPU, which they switch off when no secure inference is running. They list the prototype’s limits themselves, a development board rather than a phone, Llama 3 and Qwen3 series models only, no GPU in the pipeline, and an NPU runtime slow enough that computation becomes the bottleneck, so a faster NPU would widen their lead. Side-channel attacks, physical attacks and denial of service are out of scope, and the paper names no code release. Source: https://arxiv.org/abs/2603.09046 Read the article: https://llmobile.kavents.com/ticker/flexserve/ResearchArmNPUMemoryLlamahttps://llmobile.kavents.com/ticker/qwen-3-5/https://llmobile.kavents.com/ticker/qwen-3-5/Mon, 02 Mar 2026 10:00:00 +0100Alibaba’s Qwen team released Qwen3.5-0.8B and Qwen3.5-2B on March 2, 2026, together with 4B and 9B versions, according to the Qwen release notes. Both are dense vision-language models with a 262,144-token context and Apache 2.0 weights. The model card states that the released files work with Hugging Face Transformers, vLLM, SGLang and KTransformers, and that in light of the parameter scale the intended uses are prototyping, task-specific fine-tuning and other research or development purposes. The benchmark table in that model card puts Qwen3.5-2B at 55.3 on MMLU-Pro with reasoning switched off, where the same table gives 40.2 for last generation’s Qwen3-1.7B and 69.6 for the larger Qwen3-4B-2507, and Qwen3.5-0.8B at 29.7. With reasoning on, Alibaba reports 66.5 for the 2B and 42.3 for the 0.8B on the same test, and 51.6 against 11.9 on GPQA, a set of graduate-level science questions. The same table has Qwen3-1.7B ahead of both new sizes on IFEval, which checks whether a model obeys explicit formatting instructions, at 68.2 against 61.2 for the 2B and 52.1 for the 0.8B with reasoning off. Alibaba ran all of these figures itself. Both sizes ship a vision encoder in the same checkpoint rather than as a separate release, and Alibaba’s vision table gives Qwen3.5-2B 64.2 on MMMU, a college-level image question set, above the 61.4 it lists for the previous Qwen3-VL-2B, with 49 for Qwen3.5-0.8B. The two models stack 24 layers in which three linear-attention blocks, whose memory cost does not grow with the sequence, alternate with one full-attention block. Input embedding and output layer are tied, so a single 248,320-entry matrix does both jobs, and Alibaba puts language coverage at 201 languages and dialects. Alibaba published the weights on Hugging Face and ModelScope under Apache 2.0, and its MNN team put 4-bit exports of both sizes up the same day. The 0.8B build carries a 470 MB language weight file plus 63 MB for the vision encoder, and the 2B build 1.18 GB plus 196 MB, against 1.75 GB and 4.55 GB for the full-precision checkpoints. For other local runtimes the Qwen repository points readers to llama.cpp, which it says supports the series for text and vision, and to mlx-lm and mlx-vlm on Apple silicon, without Alibaba publishing GGUF or MLX builds of these sizes itself. Source: https://huggingface.co/Qwen/Qwen3.5-2B Read the article: https://llmobile.kavents.com/ticker/qwen-3-5/AlibabaQwenOpen weightsQuantisationBenchmarkshttps://llmobile.kavents.com/ticker/clawmobile/https://llmobile.kavents.com/ticker/clawmobile/Thu, 26 Feb 2026 13:34:00 +0100Seven researchers at MBZUAI, City University of Hong Kong and one working independently published ClawMobile on February 26, 2026, an agent runtime that runs on the Android phone itself with no tethering to a host machine. The model is not on the device. The authors state that ClawMobile runs locally while model inference is performed remotely, and all agents in their tests use GPT-5.2, so the split is an on-device runtime driving a cloud model. Across six real-life phone tasks the authors report ClawMobile completing all six at 100 percent, against 33 to 100 percent for the UI agent they compare against. What separates it from screen-reading GUI agents is that the reasoning loop never touches the screen. The orchestrator issues tool calls to control backends instead, and the authors describe a deterministic-first policy in which the runtime first checks its memory for a structured interface, reaching for ADB system commands and the Termux hardware API before anything visual. A semantic UI agent is invoked only when a step genuinely depends on reading dynamic interface state, and raw taps on interface elements sit below that as a fallback. After every action the runtime re-queries device state to confirm the step landed, which the authors contrast with inferring execution success purely through model reasoning. Architecture diagram: Du et al., CC BY 4.0. The tests ran on a Google Pixel 9 on Android 16, with DroidRun as the UI agent baseline and a stripped ClawMobile without DroidRun as a second baseline, all three on GPT-5.2 and scored by human annotators. In the authors’ table DroidRun reaches 100 percent on switching the system to dark theme and on playing a YouTube video, 85 percent on posting a YouTube comment, 73 percent on a Chrome search and on a cross-app task that searches football results and writes them into Notes, and 33 percent on installing an app from the Play Store, where ClawMobile scores 100 percent throughout. The authors put the cost of that at 57.5 seconds slower per task on average, with ClawMobile taking 21 seconds for the dark theme switch and 235 seconds for the YouTube comment. Their deterministic-only variant hits the 600-second timeout on both YouTube tasks. The authors name the remote model as the main open problem, writing that fully remote inference introduces network latency and privacy concerns and calling on-device inference a promising direction they have not taken. They also flag that serialising full UI trees or screenshots into model context can dominate token use and latency on mobile, and present the results as preliminary, six tasks on one device. The code is open source on GitHub, built on the OpenClaw agent framework with a Telegram bot as the chat channel, and the paper was accepted at EuroMLSys 2026 under a Creative Commons Attribution 4.0 license. Source: https://arxiv.org/abs/2602.22942 Read the article: https://llmobile.kavents.com/ticker/clawmobile/AgentsAndroidPixelResearchOpen sourcehttps://llmobile.kavents.com/ticker/apple-foundation-models-sdk/https://llmobile.kavents.com/ticker/apple-foundation-models-sdk/Wed, 25 Feb 2026 20:47:00 +0100Apple published python-apple-fm-sdk, Python bindings for its Foundation Models framework, with a first beta release on February 25, 2026 according to the repository’s release history. The package reaches the on-device model at the core of Apple Intelligence from Python on macOS, and installs with pip install apple-fm-sdk. The Python API follows the Swift one. SystemLanguageModel reports whether the model is available and, if not, why, while LanguageModelSession holds a conversation and its async respond call returns text or streams it. A @fm.generable decorator marks a Python class for the model to fill in, and fm.guide adds per-field constraints such as a numeric range, as the README shows with a Cat class whose age is bounded to 0 through 20. Tool calling and session transcripts are exposed as well. The bindings do not reimplement inference. The repository carries a Swift package named foundation-models-c that wraps the framework behind a C interface and builds it as a dynamic library, which the Python layer then calls. Apple writes in the documentation that the bindings run the Swift framework underneath, so evaluations reflect real on-device performance and behaviour, and its evaluation guide notes that inference calls are processed one at a time rather than in parallel at the macOS hardware level. Apple lists macOS 26.0 or later, Xcode 26.0 or later with its agreement accepted in the Xcode app, Python 3.10 or later, and Apple Intelligence switched on. PyPI carries only source distributions for the package, so the Swift component is compiled locally during installation, which is what the Xcode requirement covers. Apple Intelligence on the Mac runs on Apple silicon machines only. The code is under the Apache 2.0 license and the package metadata on PyPI classifies it as alpha. Apple pushed 0.1.0 to PyPI on March 8, 2026, added the Attachment API for sending images alongside text in 0.2.0 on June 8, 2026 for WWDC 2026, and exposed the model’s context size and the token count of a given input in 0.2.1 on June 29, 2026. The README states the project is not yet taking contributions. Source: https://github.com/apple/python-apple-fm-sdk Read the article: https://llmobile.kavents.com/ticker/apple-foundation-models-sdk/AppleDeveloper toolsApple SiliconOpen sourcehttps://llmobile.kavents.com/ticker/off-grid-offline-ai-app/https://llmobile.kavents.com/ticker/off-grid-offline-ai-app/Sat, 14 Feb 2026 23:39:00 +0100Ali Chherawalla has posted Off Grid to Show HN, an Android and iOS app that runs text generation, image generation, vision and speech recognition entirely on the device. The submission reached 124 points and 66 comments. The repository is licensed under MIT. The app bundles llama.cpp for text, Stable Diffusion for images and whisper.cpp for speech to text. It accepts any GGUF file and lists Qwen 3, Llama 3.2, Gemma 3 and Phi-4 among supported models; vision runs on SmolVLM, Qwen3-VL or Gemma 3n. It also connects to OpenAI-compatible servers on a local network, and builds a project knowledge base from PDFs chunked and embedded on-device with a bundled MiniLM model into SQLite. The README reports 15 to 30 tokens per second on flagship CPUs and 20 to 40 tokens per second through Adreno GPUs via OpenCL on a Snapdragon 8 Gen 2 or newer. Image generation is given as 5 to 10 seconds on the Hexagon NPU and vision inference at about 7 seconds. The NPU path is marked experimental: it accelerates only Q4_0 and Q8_0 quantisation, and K-quants fall back to the CPU. Comments disputed whether models of that size are usable. One reader wrote: Ive tried a lot of self hosted models at a lot of sizes; those small models are not good enough, and do not have a context long enough to be useful for most everyday operations. Chherawalla replied that “you don’t need claude level performance for a lot of day to day tasks” and that Qwen3 8B, or 4B quantised, is “actually quite good”. Other threads compared the app with PocketPal AI, SmolChat and Google’s AI Edge Gallery, asked for an F-Droid build, and reported that image generation required network access on GrapheneOS. A reader on a Samsung device reported the interface did not move for the on-screen keyboard; the developer shipped a fix the same day. Source: https://news.ycombinator.com/item?id=47019133 Read the article: https://llmobile.kavents.com/ticker/off-grid-offline-ai-app/AndroidiOSOpen sourcellama.cppImage generationhttps://llmobile.kavents.com/ticker/intelligence-per-watt/https://llmobile.kavents.com/ticker/intelligence-per-watt/Tue, 11 Nov 2025 07:33:00 +0100Researchers at Stanford University and Together AI profiled more than 20 local models on 1M real queries and report that at least one model with 20B or fewer active parameters answered 88.7% of them correctly. The best single model in their set, GPT-OSS-120B, reached 71.4% on its own, averaged across the four query collections. Their 2025 efficiency figure comes from that model running on an Apple M4 Max. The paper proposes intelligence per watt, mean task accuracy divided by the mean power the accelerator draws while answering, so a pairing that gets more answers right on the same power budget scores higher. The best local pairing each year moved from 7.92×10⁻⁴ accuracy per watt for Mixtral 8x7B on an NVIDIA Quadro RTX 6000 in 2023, to 1.80×10⁻³ for Llama 3.1 8B on an RTX 6000 Ada in 2024, to 4.18×10⁻³ for GPT-OSS-120B on an Apple M4 Max in 2025, which the authors put at a 5.3 times gain over two years. Query coverage for that single best pairing rose from 23.2% to 71.3% across the same three years. A companion metric divides accuracy by the energy a query consumes instead, so slow generation counts against a device. Diagram: Saad-Falcon et al., Figure 1 of the paper, licensed CC BY 4.0. The hardware panel separates the accelerators the authors treat as local from the cloud parts. The 1M queries come from four collections. The team took 500K prompts from WildChat, a month of real ChatGPT traffic, 500K from NaturalReasoning, 12K from MMLU Pro and 26.5K from SuperGPQA, all single-turn. WildChat has no ground truth, so GPT-4o judged whether a local model’s answer matched one written by Qwen3-235B, while the other three collections supplied reference answers. Every run used a batch size of 1, one query at a time. The hardware the authors call local is desktop and workstation class rather than mobile. Their 2025 numbers ran on an Apple Mac Studio with an M4 Max, 128 GB of unified memory and 480 W of continuous power, and the 2023 and 2024 points used NVIDIA workstation cards. No phone, tablet or wearable was profiled. Running the same Qwen3 models, the M4 Max reached 1.40 times lower accuracy per watt than an NVIDIA B200 and 1.6 to 2.3 times lower accuracy per joule, a gap the authors read as headroom for local accelerator design. For the savings case the team simulated 80.2M queries over 24 hours split between four small models on M4 Max machines and Qwen3-235B in the cloud. Perfect routing to the smallest capable model cut energy by 80.4% against sending everything to the frontier model, and a router that picks correctly 80% of the time cut it by 64.3%, with the perfect case described as a theoretical maximum. The authors note that coverage falls in technical domains such as architecture and engineering, that GPT-OSS-120B answered 88.9% of WildChat queries against 64.9% of NaturalReasoning ones, and that they release their profiling harness so the measurements can be repeated on new hardware. Source: https://arxiv.org/abs/2511.07885 Read the article: https://llmobile.kavents.com/ticker/intelligence-per-watt/ResearchBenchmarksApple SiliconNvidiaQwenhttps://llmobile.kavents.com/ticker/mobilellm-pro/https://llmobile.kavents.com/ticker/mobilellm-pro/Mon, 10 Nov 2025 06:28:00 +0100Meta Reality Labs published the MobileLLM-Pro technical report on November 10, 2025, describing a 1.08B-parameter language model built for phones and wearables. The model handles a context window of 128,000 tokens, four times what Gemma 3 1B and Qwen 3 0.6B accept. Quantising weights and embeddings to 4 bits takes the file from 2.2 GB in bf16 down to 590 MB for the CPU build and 720 MB for the accelerator build, which cannot share embedding weights. Meta exported the model through ExecuTorch and measured it on a Samsung Galaxy S25 CPU with the XNNPACK backend and on a Samsung Galaxy S24 Hexagon Tensor Processor. On a 2,000-token prompt the paper reports 8.9 s of prefill on the CPU against 2.0 s on the Hexagon unit, and decode speeds of 33.6 tok/s and 31.6 tok/s while generating 1,000 tokens. At 8,000 tokens prefill climbs to 63.5 s on the CPU and 9.8 s on the Hexagon unit, decode falls to 19.7 and 22.8 tok/s, and the key-value cache grows from 14 MB to 40 MB. The architecture puts 30 layers of width 1280 behind 20 attention heads and 4 key-value heads, and shares one weight matrix between the input embedding and the output head, which the authors say saves 260M parameters, about a quarter of the model. Three out of every four layers use local attention, comparing each token only against the 512 most recent ones, with a global layer every fourth one and at both ends. Meta’s model card credits that interleaving with cutting the key-value cache at 8,000 tokens from 117 MB to 40 MB and with 1.8 times lower prefill latency than global attention in every layer. Meta trained the model by distilling the logits of Llama 4 Scout under a KL divergence loss across three pretraining phases on roughly 1.64T tokens of open data, the second of them a step the authors call implicit positional distillation, which stretches the context window to 128k by having the teacher pass on the ability rather than feeding the student long documents. Meta puts the base model 5.7 and 7.9 points above Gemma 3 1B and Llama 3.2 1B on average across the benchmarks its model card lists, and the paper’s table has it at 76.24 on BoolQ against 63.20 and 62.51, at 52.62 on ARC-Challenge against 38.40 and 38.28, and at 100% needle-in-a-haystack retrieval against 96.80 for Llama 3.2 1B. The instruction-tuned version scores 59.8 on HumanEval against 41.5 and 37.8, 44.8 on MMLU against Gemma’s 29.9 and Llama’s 49.3, and 62.0 on IFEval against Gemma’s 80.2. In Meta’s own human evaluation over 100 prompts per task, raters preferred Gemma 3 1B for summarisation 51 to 47 and for rewriting 49 to 45, while MobileLLM-Pro led on recall 47 to 30. Meta released four checkpoints on Hugging Face, the base model, the instruction-tuned one, and two quantisation-ready base checkpoints produced by quantisation-aware training with self-distillation from the full-precision model. One uses 4-bit group-wise weights at group size 32 with 8-bit dynamic activations and an 8-bit key-value cache for CPU backends, the other 4-bit channel-wise weights for the Apple Neural Engine and Qualcomm’s Hexagon processor, which handle group-wise quantisation poorly or not at all. Needle-in-a-haystack retrieval holds at 96.44% and 98.67% against 100% at full precision. The weights sit behind a request form and carry the FAIR Noncommercial Research License. Source: https://arxiv.org/abs/2511.06719 Read the article: https://llmobile.kavents.com/ticker/mobilellm-pro/MetaResearchDistillationQuantisationAndroidhttps://llmobile.kavents.com/ticker/confidant-collaborative-training/https://llmobile.kavents.com/ticker/confidant-collaborative-training/Mon, 03 Nov 2025 10:00:00 +0100Researchers at Zhejiang University, Zhejiang University of Technology and Hangzhou Dianzi University published Confidant in the ACM MobiCom 2025 proceedings on November 3, 2025, a day before the conference opened in Hong Kong. The framework fine-tunes a language model across several off-the-shelf devices instead of one, and the authors fine-tuned Phi2-2.7B on the Alpaca instruction dataset in 40.1 hours using three of them. Their write-up of that run names a Redmi K50 phone, a Core i7-13650HX laptop and a MacBook Pro with an M1 Pro, while the paper’s abstract and conclusion describe the same result as three mobile devices. The same trio fine-tuned BERT-Base in 2.2 hours. Confidant cuts a model into consecutive slices, gives each device one slice, and passes activations from one device to the next, so no device ever holds the whole model. Batches flow through that chain as a pipeline, with each device starting a new batch while the devices behind it are still working on earlier ones. The device that holds the training data acts as coordinator and recalculates where the cuts fall during training, using each participant’s measured compute time, free memory and bandwidth, and it discards any split whose slice would not fit in a device’s memory. Inside a device, a second scheduler profiles the CPU and the GPU on varying numbers of attention heads and gives each processor as many heads as it can finish in roughly the same time, which is what the authors credit for running up to 1.94 times faster than a capacity-aware pipeline baseline they implemented themselves, and up to 3.82 times faster than the best single device, both on GPT-2 Medium. Diagram: Chen et al., Figure 3 of the paper, licensed CC BY 4.0. Splitting the model is what brings the footprint per device down. The paper’s memory table puts fine-tuning BERT-Base at a batch size of 8 at 5.5 GB on a single Redmi K50, against a per-device average of 3.17 GB across three devices and 2.43 GB across four. Larger models leave no single-device figure to compare with, because a Redmi K50 fits only 1 of Phi2-2.7B’s 32 transformer blocks at a batch size of 4, and the per-device averages of 15.31 GB for Phi2-2.7B and 32.07 GB for LLaMA3-8B across three devices are carried by the two laptops rather than the phone. A separate table puts the memory PyTorch needs to fine-tune Phi2-2.7B at a batch size of 8 at 46.5 GB, next to the 8 GB to 12 GB of the four phones it lists. The paper’s prose says Confidant almost does not cause any accuracy drop, while its own model performance table puts the three-device runs at a test loss of 0.66 on Phi2-2.7B and 3.55 on LLaMA3-8B, against 0.52 and 2.86 for fine-tuning the same models the conventional way on a single machine, where a lower loss is better. The four-device runs land at 0.76 and 2.85 on that same comparison, and BERT-Base token classification accuracy holds at 97.66% against 97.70%. Keeping data on the device is the stated motivation. The authors write that training with Confidant does not require sharing raw data and that devices exchange only intermediate outputs and model gradients, which they present as the privacy argument for their two scenarios, a household assistant fine-tuned on family data during idle hours and a conference assistant fine-tuned on attendees’ phones. They also state that fine-tuning on a single cloud server would be faster and more straightforward, and that their aim is to use idle mobile hardware instead. The Android side is built on Alibaba’s MNN framework with laptops running PyTorch, about 10,900 lines of code in total, the paper is published under a Creative Commons Attribution 4.0 license, and the authors wrote that they planned to release the code in late 2025. Source: https://dl.acm.org/doi/10.1145/3680207.3723487 Read the article: https://llmobile.kavents.com/ticker/confidant-collaborative-training/ResearchMemoryPhiLlamaAndroidhttps://llmobile.kavents.com/ticker/executorch-1-0/https://llmobile.kavents.com/ticker/executorch-1-0/Wed, 22 Oct 2025 18:00:00 +0200The PyTorch team released ExecuTorch 1.0 on October 22, 2025, the generally available version of its runtime for PyTorch models on phones, laptops and embedded devices. It takes a model straight from PyTorch without converting it to ONNX or TFLite and without rewriting it in another language, which the team says removes a common source of numerical mismatches and lost debug information. The beta arrived in October 2024. Diagram: PyTorch documentation. Five backends moved from beta to production status. XNNPACK with Arm Kleidi covers CPU, Apple Core ML covers Apple silicon, the Qualcomm AI Engine delegate covers the Hexagon NPU, and Arm Ethos-U and Vulkan cover NPU and GPU. New backends arrived for Arm VGF, the NXP eIQ Neutron NPU, Samsung Exynos NPU and GPU, and Intel OpenVINO. Meta says features built on ExecuTorch already run on Instagram, WhatsApp, Messenger and Facebook. Anuj Kumar, Director at Reality Labs, describes what it drives in Meta’s wearables, including the Ray-Ban Display glasses with EMG band. Advanced AI features like speech recognition, motion sensing, and computer vision run on these devices by leveraging cutting-edge hardware. Lysandre Debut, Chief Open-Source Officer at Hugging Face, says more than 80 percent of the most downloaded edge-friendly language models on Hugging Face run on ExecuTorch out of the box. The team validated coverage across object detection, depth, OCR, speech recognition and segmentation as well as text and multimodal models such as Voxtral for audio and Gemma 3 for image and text input. The main branch also allows embedding ExecuTorch into native C++ desktop applications. Source: https://pytorch.org/blog/introducing-executorch-1-0/ Read the article: https://llmobile.kavents.com/ticker/executorch-1-0/MetaPyTorchDeveloper toolsNPUOpen sourcehttps://llmobile.kavents.com/ticker/lm-meter/https://llmobile.kavents.com/ticker/lm-meter/Tue, 07 Oct 2025 19:05:00 +0200Researchers at Georgia State University and Toyota InfoTech Labs published lm-Meter, a latency profiler that runs inside the inference engine on the phone and splits each generation into embedding, prefill, decode, softmax and sampling. Measuring the Pythia models on a Google Pixel 8 Pro, they report that scaling from 70M to 1.4B parameters raises prefill latency from 0.012 s to 1.9 s per input token, a 158x slowdown, while decode latency per output token grows from 0.015 s to 0.15 s, a 10x slowdown. The authors write that this inverts the server picture, where decode is usually the limiting phase for single-request inference. Below the phase level the profiler times individual GPU kernels through OpenCL event timestamps, which give queue, submit, start and end times without access to the closed-source driver. Running a 4-bit quantised Gemma-2-2B-it on a Pixel 8 Pro, the authors report that fused matrix-multiplication kernels dominate a decode step and that the GPU sits idle for more than 21% of it, the second-largest contributor to the step, which they attribute to host-side data preparation and I/O stalls. The paged attention kernel that scans the growing key-value cache is the only one whose cost rises with position in the sequence, climbing from roughly 0.2 ms to about 0.8 ms per token over 250 decode steps, and idle time drops from about 21% to 12% when the model generates 256 tokens instead of 16. Whether those measurements mean anything depends on what the profiler itself costs. lm-Meter sits in the MLC LLM runtime and TVM in about 3,500 lines of code and needs no host machine attached, and under the Powersave CPU governor, the most constrained setting they tested, the authors measure a throughput loss of 2.58% in prefill and 0.99% in decode. They put the same figures for MELTing Point, the on-device profiler they compare against, at 22% for prefill and more than 93% for decode. Checked against traces from Android GPU Inspector, they report end-to-end phase accuracy of at least 99.99% and mean kernel-level accuracy of 96.82% on the Pixel 8 Pro and 96.61% on a Pixel 7. The code is on GitHub under the MIT license, with the MLC LLM path released for Android GPUs through OpenCL and support for llama.cpp, vLLM, iOS Metal and Nvidia Jetson listed as unfinished. The work was accepted to the ACM/IEEE Symposium on Edge Computing 2025 and funded by Toyota Motor North America. The measurements come from three phones, the Pixel 8 Pro, Pixel 7 and Pixel 6, and the authors state that other edge platforms such as Jetson boards and Intel NPUs may show different bottlenecks. Source: https://arxiv.org/abs/2510.06126 Read the article: https://llmobile.kavents.com/ticker/lm-meter/ResearchBenchmarksDeveloper toolsMLC LLMPixelhttps://llmobile.kavents.com/ticker/mobilellm-r1/https://llmobile.kavents.com/ticker/mobilellm-r1/Fri, 12 Sep 2025 12:00:00 +0200Meta released MobileLLM-R1 on September 12, 2025, a family of reasoning models at 140M, 360M and 950M parameters that write out a chain of intermediate steps before giving an answer instead of replying directly. Meta reports the 950M model scoring 74.0 on the MATH500 maths benchmark and 19.9 on LiveCodeBench v6, against 73.0 and 14.9 for Qwen3-0.6B. The technical report followed on September 29, 2025. The three models saw 4.2T training tokens, which Meta puts at 11.7% of the 36T tokens behind Qwen3-0.6B. Pre-training ran in two phases of 2T tokens each, resampled from a curated pool of roughly 2T tokens of open datasets, followed by two mid-training phases of 100B tokens that distil from Llama-3.1-8B-Instruct as a teacher, then supervised fine-tuning on reasoning traces. Meta uses the result to argue that reasoning does not require the corpora above 10T tokens that recent models have been trained on. For the post-trained models Meta’s table puts MobileLLM-R1-950M at 74.0 on MATH500, 67.5 on GSM8K, 15.5 on AIME 2024 and 19.9 on LiveCodeBench v6, where Qwen3-0.6B reaches 73.0, 79.2, 11.3 and 14.9 and DeepSeek-R1-Distill-Qwen-1.5B reaches 83.2, 77.3, 29.1 and 19.9. Among the models whose weights, data and recipe are all public, the same AIME figure compares with 0.6 for OLMo-2 1.48B and 0.3 for SmolLM2-1.7B. On code the 950M base model scores 46.3 on HumanEval against 30.5 for Qwen3-0.6B-Base, and the 140M base model reaches 15.9 where SmolLM2-135M scores 0.0. Meta profiled the models through ExecuTorch on a Samsung Galaxy S22 with 8 GB of RAM, quantised to 4-bit weights with 8-bit dynamic activations. The 140M model generated 129.67 tok/s at a 1k context and 79.71 tok/s at 32k, the 950M model 31.05 tok/s at 1k before running out of memory at 16k, and Llama-3.2-1B 28.71 tok/s at 1k with the same memory failure. The architecture follows Meta’s earlier MobileLLM work and Llama 3.2, sharing one weight matrix between the input embedding and the output layer, and the post-trained models take a 32k context. Meta released the weights on Hugging Face together with the training code, the data sources and the data mixing ratios on GitHub, under the FAIR Noncommercial Research License, which covers research, education and analysis but not commercial use. Access to the model repositories is gated behind a form. Update, November 24, 2025. Meta released MobileLLM-R1.5 at the same three sizes, fine-tuned from the R1 models with one further round of on-policy distillation, where the student answers first and the teacher, here Llama-3.1-Nemotron-Nano-4B-v1.1, grades its own words back to it token by token. Meta reports the 950M version at 86.6 on MATH500, 82.6 on GSM8K, 39.9 on AIME 2024 and 29.1 on LiveCodeBench v6, up from 74.0, 67.5, 15.5 and 19.9 for MobileLLM-R1-950M, and says it beats DeepSeek-R1-Distill-Qwen-1.5B on every maths and coding benchmark it ran. At 360M Meta puts MATH500 at 63.4 against 28.4 and GSM8K at 52.8 against 24.5. Source: https://arxiv.org/abs/2509.24945 Read the article: https://llmobile.kavents.com/ticker/mobilellm-r1/MetaResearchBenchmarksDistillationAndroidhttps://llmobile.kavents.com/ticker/a19-pro-neural-accelerators/https://llmobile.kavents.com/ticker/a19-pro-neural-accelerators/Tue, 09 Sep 2025 20:00:00 +0200Apple announced the A19 Pro alongside the iPhone 17 Pro on September 9, 2025. Its 6-core GPU carries Neural Accelerators built into each core, which Apple says work together with the 16-core Neural Engine to power AI models, graphics and games. The company states that the chip enables running large local language models on the phone. The A19 Pro also has a 6-core CPU that Apple calls the fastest in any smartphone, along with a larger cache and more memory than the A18 Pro. Combined with a vapor chamber cooling system, Apple puts sustained performance at up to 40 percent above the previous generation. Image: Apple. The vapor chamber Apple credits for the sustained performance gain. Source: https://www.apple.com/newsroom/2025/09/apple-unveils-iphone-17-pro-and-iphone-17-pro-max/ Read the article: https://llmobile.kavents.com/ticker/a19-pro-neural-accelerators/AppleiPhoneChipsApple SiliconNPUhttps://llmobile.kavents.com/ticker/shadownpu/https://llmobile.kavents.com/ticker/shadownpu/Fri, 22 Aug 2025 09:41:00 +0200Researchers from Peking University and BUPT report up to 4.5 times faster end-to-end inference on a Xiaomi 14 with a Snapdragon 8 Gen 3, and 3.5 times faster on average for the attention kernel alone, with a peak of 6.9 times. The baseline is the arrangement most on-device frameworks use today, float32 full attention running on the CPU or GPU while the rest of the model sits on the NPU. The authors measured Qwen2-0.5B, Qwen2-1.5B, PhoneLM-0.5B and PhoneLM-1.5B on summarisation and two mobile agent tasks, and gave their own system only one of the phone’s middle CPU cores. Against the native attention of llm.npu, which uses four CPU cores or the GPU, they report up to 3.0 times lower latency from that one core. The paper is due at MobiSys 2026. The starting problem is that attention keeps falling back off the NPU. Attention multiplies three activation tensors together, and mobile NPUs compile a static graph that fixes one quantisation scale factor for a whole tensor, so the authors measure an average 18 percentage point accuracy drop when the operator runs in INT8 on the NPU. Their trade is to stop asking the NPU for the answer and ask it only for the ranking. The NPU computes the query-key scores in INT8 purely to find which tokens matter, which the authors report recovers over 99 percent of the positions a float run would pick, since picking the largest values needs relative magnitudes rather than exact ones. The CPU then computes real attention in float over roughly 20 percent of the tokens and throws the other 80 percent away, at an average accuracy cost of 0.4 percentage points against the float baseline. Each attention head gets its own sparsity ratio, fixed by an offline profiling run the paper puts at about five minutes on a single A100. On a Redmi K60 Champion Edition with a Snapdragon 8 Gen 2 the authors read battery voltage and current from the kernel and report a single attention kernel at 1024 tokens dropping from 8.59 J to 1.12 J for PhoneLM-1.5B, a factor of 7.66. Speedups on that older phone are smaller, 2 times, 1.25 times and 1.22 times across the three datasets, which the authors attribute to its Hexagon V73 NPU against the V75 in the Xiaomi 14. The paper reports no memory footprint figures. It does report that end-to-end inference time on the Xiaomi 14 stayed between 14.5 and 15.5 seconds while the phone was also taking photos, playing video, playing music or running a shopping app, since the LLM work is not competing for the cores those apps use. Diagram: Yin et al. The authors state a prototype of more than 10,000 lines of C++ and Python, built on Qualcomm’s QNN and Hexagon SDK for the NPU side and integrated into llm.npu, which they call the only fully open-source framework running on NPUs. The paper names no repository and announces no artefact release, and the title calls the system ShadowNPU while the text throughout calls the module shadowAttn. The arXiv listing carries a CC BY 4.0 license. For a hands-on point of comparison on the same silicon family, a user test of llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone put generation at about the same speed as the chip’s CPU, without the heat. Source: https://arxiv.org/abs/2508.16703 Read the article: https://llmobile.kavents.com/ticker/shadownpu/QualcommNPUResearchQwenAndroidhttps://llmobile.kavents.com/ticker/pd-device-cloud-split/https://llmobile.kavents.com/ticker/pd-device-cloud-split/Tue, 12 Aug 2025 17:56:29 +0200Thirty researchers at Huawei Technologies published P/D-Device on August 12, 2025, a scheme that runs a single language model request across a cloud data centre and a phone at the same time instead of choosing one of them. The cloud reads the prompt and writes only the first few tokens, then the device carries the answer to the end. The authors measure average time to first token falling by at least 60 percent against running the whole request on the device, and cloud throughput rising by 1.6 to 15 times, 7.6 times on average, against letting the cloud generate a full 200-token answer. The split follows from the two phases of inference wanting different hardware. Prefill, where the model reads the prompt before it writes anything, is compute-bound, and the authors report on-device time to first token climbing steeply with prompt length, into seconds or tens of seconds once prompts run to several thousand tokens. Decoding, where the model emits one token at a time, behaves differently, because a phone serves one user while a cloud instance serves a large batch, so the authors find the two roughly matched at tens of milliseconds per output token. Decoding is also what occupies a cloud instance for tens of seconds per request, so the scheme cuts the cloud off after a set number of tokens and hands the rest of the answer to the device. The device displays the cloud’s first token the moment it arrives, so the wait the user actually perceives is the cloud’s prefill, which the authors put at hundreds of milliseconds. Behind that display the device runs its own prefill while a speed controller releases the next tokens from the cloud at a deliberately slowed pace, matched to human reading speed, until the device has caught up and generates the rest itself. During its prefill the cloud also scores the prompt using the attention weights it has already computed, picks the sentences that matter most, and returns the selection as a compressed mask so the device prefills a shorter prompt, with a quarter of the prompt kept by default. What crosses the network is text, never model state. The device sends the prompt as raw text and the cloud returns the first token, the mask and a token budget, which the authors cap at hundreds of kilobytes for an 8k prompt, while the KVCache, the gigabyte-scale intermediate state that other cloud-device schemes ship back and forth, is too large to move in time and is never transferred at all. On the prompt text itself the authors state that the request goes out only under user and app authorisation and after sensitive information such as phone numbers and real names has been removed or substituted, and they cite Apple’s Private Cloud Compute as the class of guarantee they have in mind, without describing an implementation or measuring one. The archive’s post on Flower Intelligence covers the reverse arrangement, where the device runs first and the remote service stays off unless the app turns it on. Diagram: Jin et al. The cloud scores the prompt during its own prefill and returns the selection as a mask alongside the first token. The prototype runs cloud inference on Huawei Ascend NPUs and device inference on Kirin SoCs, tested on a phone and a tablet, with every chip and model anonymised as Type 1, Type 2, Model 1 and Model 2 throughout the paper. On LongBench with the same model on both sides, the authors’ own table gives their refinement an average of 26.64 on single-document QA against 25.89 for the unrefined on-device baseline, and 26.83 against 22.96 on multi-document QA. The authors name the costs themselves, a quality score around 85 percent of cloud-only inference when the cloud runs the larger model, refinement adding one or two hundred milliseconds at an 8k prompt and several hundred under load, and the smoothed display running slightly above their own tolerable threshold of about a hundred milliseconds per token once prompts pass 8k. The paper, on arXiv under a CC BY 4.0 license, further states that connections were tested over Wi-Fi with an IP whitelist rather than LTE, that collaboration across different NPU types is still being explored, and that the device supports prompts of several thousand tokens against hundreds of thousands in the cloud. Source: https://arxiv.org/abs/2508.09035 Read the article: https://llmobile.kavents.com/ticker/pd-device-cloud-split/ResearchHuaweiNPUBenchmarkshttps://llmobile.kavents.com/ticker/demystifying-slms-edge/https://llmobile.kavents.com/ticker/demystifying-slms-edge/Sun, 27 Jul 2025 09:00:00 +0200Ten researchers at Beijing University of Posts and Telecommunications, Peng Cheng Laboratory, the University of Cambridge, Xiaomi, Helixon Research and Flower Labs collected 68 small language models between 100M and 5B parameters, released by 24 organisations between OPT in May 2022 and Llama 3.2 in September 2024, and measured both their accuracy and their cost on edge hardware. In Demystifying Small Language Models for Edge Deployment, presented at ACL 2025, Microsoft’s Phi-3 and Phi-3.5-mini sit highest on the capability chart at about 70 percent average accuracy over 10 reasoning and knowledge datasets, level with the Llama 3.1 line the authors draw for comparison. The runtime half of the study covers the 20 models that llama.cpp supports, quantised to 4-bit and run with a 50-token prompt and 50 generated tokens. Chart: Lu et al., licensed CC BY 4.0. Accuracy is averaged over the 10 datasets with math excluded, and circle size is proportional to model size. Parameter count predicts on-device latency poorly. The measurements put OpenELM-3B at 9.4 percent more parameters than Phi-2 but 9.9 percent faster to the first token, and Qwen1.5-0.5B at 25.4 percent more parameters than Qwen2-0.5B yet 31.9 percent faster to the first token and 11.8 percent faster per decoded token. Qwen1.5-1.8B carries 30.5 percent more parameters than Pythia-1.4B and still reaches the first token 18.5 percent sooner. The authors trace the gap to how a model spends its parameter budget, with Qwen2 sharing its input embedding and output layer and putting the saved weights into wider attention and feed-forward layers, which costs compute in the prefill stage. Memory ran from 275 MB to 2456 MB across the 20 models, measured with llama.cpp on a Jetson Orin NX at a fixed 2048-token context. Vocabulary size moves that figure as much as parameter count does, and the paper reports Bloom-560M needing 492 MB of compute buffer for its 250,880-token vocabulary, 3.5 times what OpenELM-1.1B needs with 32,000 tokens. OpenELM-1.1B uses 23.8 percent less memory than Bloom-560M despite 32.3 percent more parameters, and OpenELM-3B 13.8 percent less than Gemma-2B despite 21.1 percent more, which the authors put down to the small vocabulary and to grouped-query attention, where several attention heads share one set of cached keys and values. Raising each model to its own maximum context window pushes Qwen2-0.5B past 6 GB, with compute buffer and KV cache making up 85 percent of the total for the Qwen2 models. Charts: Lu et al., licensed CC BY 4.0. Measured with llama.cpp at 4-bit, and memory taken on a Jetson Orin NX at a 2048-token context. Quantisation paid off in decoding rather than in prefill. Testing five settings on Phi-1.5, the authors found Q4_K_M the strongest, roughly halving per-token decode latency against FP16, while at a 50-token prompt it cut the time to the first token by only 13 percent. Q6_K and Q3_K matched or exceeded FP16 on longer prompts, which the paper puts down to the alignment and padding their irregular bit widths force. The hardware gap was wider than the quantisation gap, with the Jetson’s GPU 40 times faster than the Meizu 18 Pro’s CPU at prefill but only 1.84 times faster at decode, and the Xiaomi 12S on a Snapdragon 8 Gen 1+ the fastest of the three phones, ahead of the Pixel 7 Pro and the Meizu on its Snapdragon 888. The authors name two limits on their own work. They left math datasets out of the capability evaluation because of the gap between small models and larger ones on mathematical reasoning, and they confined the cost analysis to the 20 models llama.cpp supports so that the inference engine stayed constant, excluding every model it cannot load. The paper is free to read at the ACL Anthology under a Creative Commons Attribution 4.0 licence, and the same group had earlier measured 22 models from 0.5B to 7B on four Android phones, three of which reappear in this test bed. PalmBench adds power draw and surface temperature on comparable hardware, and Artificial Analysis has since reported quality, speed and memory together for 33 quantised models on one iPhone 17 Pro. Source: https://aclanthology.org/2025.acl-long.718/ Read the article: https://llmobile.kavents.com/ticker/demystifying-slms-edge/BenchmarksResearchllama.cppMemoryAndroidhttps://llmobile.kavents.com/ticker/apple-foundation-models-report-2025/https://llmobile.kavents.com/ticker/apple-foundation-models-report-2025/Fri, 18 Jul 2025 01:37:00 +0200Apple posted its 2025 foundation models tech report to arXiv on July 17, 2025, giving the numbers behind the roughly 3-billion-parameter on-device model it shipped at WWDC25. The report’s Table 3 puts the price of compressing that model to 2 bits per weight at 3.4 points of MMLU, from 67.8 in 16 bits down to 64.4, with IFEval falling from 85.1 to 82.3. Apple reaches 2 bits through quantisation-aware training with a learnable scaling factor per weight tensor, and reports that a balanced set of four levels at -1.5, -0.5, 0.5 and 1.5 trains more smoothly than the usual -2, -1, 0 and 1. The embedding table is quantised to 4 bits and the key-value cache to 8 bits, and low-rank adapters are fine-tuned afterwards to claw back the quality the compression costs. The server model takes a different route, compressed after training to 3.56 bits per weight with Adaptive Scalable Texture Compression, a GPU texture format whose fixed-function decode hardware on Apple GPUs unpacks the weights at no cost to the compute cores. The report also spells out the KV cache sharing the framework announcement only named. Apple splits the transformer so that Block 1 holds 62.5 percent of the layers and Block 2 the remaining 37.5 percent with its key and value projections removed, reusing Block 1’s cache instead. That cuts KV cache memory by 37.5 percent, and because Block 2 produces no keys or values, prefill skips its computation entirely, which Apple says cuts time to first token by about the same 37.5 percent. Apple rebuilt how the on-device model is trained. The company first trained a dense model on about 14T tokens, sparse-upcycled it into a 64-expert mixture-of-experts version, where only part of the model runs per token, using 1T high-quality tokens, then retrained the dense model on its last 10 percent of tokens against a distillation loss from that MoE teacher. Apple states this cut the cost of training the teacher by 90 percent and removed the need for structural pruning. The server model was trained on 8192 v5p Cloud TPU chips for 13.4T tokens. For the language expansion the report states the models are designed to support 16 languages, built on a text tokenizer grown from a 100k to a 150k vocabulary and a continued pre-training mix whose multilingual share rose from 8 percent to 30 percent. In Apple’s own human evaluation on US English text prompts, graders preferred the on-device model over Qwen-2.5-3B on 35.3 percent of prompts against 12.9 percent the other way, while against Gemma-3-4B the split was 21.0 percent to 21.9 percent. The report is on arXiv under a Creative Commons Attribution 4.0 license. Chart: Apple's own human evaluation, from the tech report. Rows marked with an asterisk were tested against the compressed on-device model. Source: https://arxiv.org/abs/2507.13575 Read the article: https://llmobile.kavents.com/ticker/apple-foundation-models-report-2025/AppleApple SiliconQuantisationDistillationBenchmarkshttps://llmobile.kavents.com/ticker/lfm2/https://llmobile.kavents.com/ticker/lfm2/Thu, 10 Jul 2025 14:00:00 +0200Liquid AI released LFM2 on July 10, 2025, three open-weight checkpoints of 350M, 700M and 1.2B parameters aimed at the CPUs inside phones, laptops and vehicles. The company reports 2x faster decode and prefill speed on CPU than Qwen3, and describes the models as built for millisecond latency and offline operation on the device rather than in a data centre. The architecture is a hybrid of 16 blocks, 10 of them gated short convolutions and 6 grouped query attention. A short convolution mixes each token with a handful of its neighbours instead of attending over the whole history, which is cheap on a phone CPU and keeps the attention cache small, while the six attention blocks preserve the recall that pure convolutions lose. Liquid AI says the layout came out of its STAR architecture search, which it ran against measured peak memory and measured prefill plus decode speed on Qualcomm Snapdragon embedded SoC CPUs rather than against proxy metrics. Liquid AI benchmarked the models on a Samsung Galaxy S24 Ultra with a Qualcomm Snapdragon SoC and on an AMD Ryzen HX370 laptop, using Q4_0 in llama.cpp and 8da4w in ExecuTorch. In its llama.cpp charts LFM2-350M clears 1,300 tokens per second of prefill and runs at roughly 190 tokens per second of decode on the Galaxy S24 at a 128-token sequence, falling to around 120 tokens per second of decode at 4,096 tokens. The company states that LFM2-700M stays faster than Qwen3-0.6B on both decode and prefill in both runtimes despite being 16 percent larger, and that the CPU gains should carry over to GPUs and NPUs once kernels are optimised. Charts: Liquid AI's own llama.cpp measurements at Q4_0 quantisation. The models are small enough to ship whole, with Liquid AI’s own Q4_0 GGUF builds at 219 MB for LFM2-350M and 696 MB for LFM2-1.2B. On Liquid AI’s internal evaluation table LFM2-1.2B scores 55.23 on MMLU and 74.89 on IFEval against 59.11 and 73.98 for Qwen3-1.7B, a model with 47 percent more parameters, while LFM2-700M reaches 49.9 on MMLU against 40.08 for Gemma 3 1B IT. Liquid AI notes that it evaluated Qwen3 in non-reasoning mode only, on the grounds that reasoning traces run past the output budgets that matter on edge devices. Chart: Liquid AI. The average covers MMLU, IFEval, IFBench, GSM8K and MMMLU. Weights are on Hugging Face, with the models also reachable through the Liquid Playground and OpenRouter, and Liquid AI points developers at llama.cpp and ExecuTorch for local testing. The licence is an open licence based on Apache 2.0 that permits academic and research use freely and commercial use by companies under $10 million in revenue, with anyone above that threshold directed to contact the company for a commercial licence. All three checkpoints were trained on 10T tokens, roughly 75 percent English, 20 percent multilingual and 5 percent code, with the context extended to 32K during pre-training. Update, September 23, 2025. Liquid AI added LFM2-2.6B, which widens the stack to 30 blocks, 22 of them convolutions and 8 attention. The company reports 82.41 percent on GSM8K and 79.56 percent on IFEval, and says the model outperforms Llama 3.2-3B-Instruct, Gemma 3 4B IT and SmolLM3-3B. It ships on Hugging Face under the same licence. Update, November 28, 2025. Liquid AI published an LFM2 technical report covering the dense 350M to 2.6B models plus an 8.3B mixture-of-experts variant with 1.5B active parameters, where only part of the model runs per token. Measuring on a Samsung Galaxy S25 with a Snapdragon 8 Elite in llama.cpp at Q4_0, it reports LFM2-350M at 1,067 tokens per second of prefill and 194.1 tokens per second of decode on a 1K-token prompt, and LFM2-2.6B at 143 and 33.8 against 57 and 17.2 for Qwen3-4B. The report names ExecuTorch, llama.cpp and vLLM as the shipped deployment packages. Source: https://www.liquid.ai/blog/liquid-foundation-models-v2-our-second-series-of-generative-ai-models Read the article: https://llmobile.kavents.com/ticker/lfm2/Liquid AILFMOpen weightsllama.cppBenchmarkshttps://llmobile.kavents.com/ticker/smollm3/https://llmobile.kavents.com/ticker/smollm3/Tue, 08 Jul 2025 12:30:00 +0200Hugging Face released SmolLM3 on July 8, 2025, a 3B model trained on 11.2 trillion tokens that takes up to 128k tokens of context, covers six languages and answers in either of two modes. It is the successor to SmolLM2, whose largest model was 1.7B with an 8k context window and, in Hugging Face’s own words, primarily understands and generates content in English. The longer context comes from an architecture change plus a separate training stage. Hugging Face removed rotary position embeddings from every fourth layer, a technique it calls NoPE, and replaced multi-head attention with grouped-query attention to shrink the key-value cache during inference. After the main pretraining run it trained the model on another 100 billion tokens in two steps, 4k to 32k and then 32k to 64k, and YaRN, a method for extrapolating past the trained length, carries it to 128k tokens at inference, though the published configuration ships set to 64k and has to be edited to go further. Multilingual text made up 12 percent of the web data at every pretraining stage, and Hugging Face lists English, French, Spanish, German, Italian and Portuguese as supported. Hugging Face’s own base-model figures put SmolLM3 at 76.2 on HellaSwag, 65.6 on ARC and 79.0 on BoolQ, ahead of both Llama 3.2 3B and Qwen2.5 3B and ahead of the larger Qwen3 4B on all three. Qwen3 4B stays in front on maths and code, at 51.2 against 46.1 on MATH and 54.9 against 30.5 on HumanEval+, and on the Ruler 64k long-context test Llama 3.2 3B leads at 72.9 against 67.9 for SmolLM3. On the Flores-200 translation benchmark SmolLM3 scores 56.3 against 55.8 for Qwen3 4B and 54.9 for Qwen2.5 3B, and its multilingual chart shows it ahead of both 3B rivals in all five non-English languages and behind Qwen3 4B in all five. Table: Hugging Face's own base-model figures. A flag in the system prompt, /think or /no_think, decides whether the instruct model reasons before it answers. Hugging Face reports the reasoning mode at 36.7 percent on the AIME 2025 maths competition against 9.3 percent without it, 41.7 against 35.7 on GPQA Diamond and 30.5 against 15.2 on LiveCodeBench v4. Qwen3 4B leads on most of the eight tests in that comparison in both modes, at 58.8 on AIME 2025 and 52.9 on LiveCodeBench, though SmolLM3 without reasoning tops it on the IFEval instruction-following test at 76.7 to 68.9. Table: Hugging Face's own instruct-model figures. Hugging Face published the recipe as well as the weights. Pretraining ran in three stages that moved web data from 85 percent to 75 and then 63 percent while code rose from 12 percent to 24 and maths from 3 percent to 13, each share drawn from named public datasets, and the company released the training configs with the exact data weights along with intermediate checkpoints. Both the base and instruct models are on Hugging Face under the Apache 2.0 license, and the model card names llama.cpp, ONNX, MLX, MLC and ExecuTorch for local inference. The 4-bit GGUF published by ggml-org on release day is 1.92 GB, against 3.27 GB for the 8-bit build for MLX and 6.15 GB for the bfloat16 weights. Diagram: Hugging Face. Source: https://huggingface.co/blog/smollm3 Read the article: https://llmobile.kavents.com/ticker/smollm3/Hugging FaceOpen weightsBenchmarksOpen sourcellama.cpphttps://llmobile.kavents.com/ticker/apple-foundation-models-framework/https://llmobile.kavents.com/ticker/apple-foundation-models-framework/Mon, 09 Jun 2025 21:00:00 +0200Apple opened its on-device foundation model to third-party apps at WWDC25 on June 9, 2025. The Foundation Models framework gives any app direct access to the roughly 3-billion-parameter model, which runs offline and costs developers nothing per call. Developers annotate Swift data structures with the @Generable macro, and the framework applies constrained decoding, which Apple calls guided generation, so the model returns those structures rather than free text. Tool calling lets the model invoke functions the app supplies. Developers who need more control can train rank 32 adapters with a Python toolkit. Apple compressed the model to 2 bits per weight using quantisation-aware training. It splits the transformer into two blocks at a 5:3 depth ratio and shares the key-value cache between them, which Apple says cuts KV cache memory use by 37.5 percent. Images are handled by a 300M-parameter ViTDet-L vision encoder. The model is designed to support 15 languages. The companion server model on Private Cloud Compute uses what Apple calls a Parallel-Track Mixture-of-Experts design, where independent transformer tracks cut how often the tracks have to synchronise with each other. Diagram: Apple. The server model, not the on-device one. Source: https://machinelearning.apple.com/research/apple-foundation-models-2025-updates Read the article: https://llmobile.kavents.com/ticker/apple-foundation-models-framework/AppleiOSDeveloper toolsQuantisationAgentshttps://llmobile.kavents.com/ticker/generative-ai-at-the-edge-queue/https://llmobile.kavents.com/ticker/generative-ai-at-the-edge-queue/Wed, 21 May 2025 18:00:00 +0200Vijay Janapa Reddi of Harvard University published an article in ACM Queue on May 21, 2025, arguing that generative models reach phones, glasses and robots only if the field stops chasing parameter counts and starts designing for deployment. He projects more than 50 billion edge devices by 2030 and writes that today’s frontier models, often hundreds of billions of parameters, cannot leave the data centre. His deployment matrix places only models under 1B parameters in the viable column for true edge hardware, puts medium models of 1B to 10B parameters within reach of high-end phones and single GPUs, and confines anything larger to servers or the cloud. Reddi grounds that in memory first. He writes that a 6B parameter model at half precision needs roughly 12 GB for weights alone, far above typical mobile RAM, that even a 1B model at about 2 GB can strain some devices, and that small models usually occupy 100 MB to 2 GB. He puts the latency bar at around 50 ms for an augmented reality translation to feel instantaneous and under a second for a dialogue assistant to feel fluid, and he notes that batching across users, which keeps cloud serving efficient, does not apply when requests arrive one at a time. On energy he cites an int8 quantised model using about five times less energy per query than an FP16 one on the same hardware, and says MLPerf Power results suggest model energy efficiency is plateauing even as hardware improves. Reddi frames the whole problem as a data-model-compute triangle, where scarce local data, limited compute and the need for compact models each worsen the others, and he argues compression alone will not close the gap, since smaller models hallucinate more and can lose their refusal behaviour when quantised hard. He proposes measuring capability and cost together through metrics such as hallucinations per watt-hour, running a model on factual question answering until it consumes 1 Wh and counting the wrong statements, and toxicity per token normalised by model size. The article ran in Queue volume 23, issue 2, and the full text is free to read on queue.acm.org. Source: https://dl.acm.org/doi/abs/10.1145/3733702 Read the article: https://llmobile.kavents.com/ticker/generative-ai-at-the-edge-queue/ResearchMemoryQuantisationNPUWearables