Mobile AI news ticker

Apple published python-apple-fm-sdk, Python bindings for its Foundation Models framework, with a first beta release on February 25, 2026 according to the repository’s release history. The package reaches the on-device model at the core of Apple Intelligence from Python on macOS, and installs with pip install apple-fm-sdk.

The Python API follows the Swift one. SystemLanguageModel reports whether the model is available and, if not, why, while LanguageModelSession holds a conversation and its async respond call returns text or streams it. A @fm.generable decorator marks a Python class for the model to fill in, and fm.guide adds per-field constraints such as a numeric range, as the README shows with a Cat class whose age is bounded to 0 through 20. Tool calling and session transcripts are exposed as well.

The bindings do not reimplement inference. The repository carries a Swift package named foundation-models-c that wraps the framework behind a C interface and builds it as a dynamic library, which the Python layer then calls. Apple writes in the documentation that the bindings run the Swift framework underneath, so evaluations reflect real on-device performance and behaviour, and its evaluation guide notes that inference calls are processed one at a time rather than in parallel at the macOS hardware level.

Apple lists macOS 26.0 or later, Xcode 26.0 or later with its agreement accepted in the Xcode app, Python 3.10 or later, and Apple Intelligence switched on. PyPI carries only source distributions for the package, so the Swift component is compiled locally during installation, which is what the Xcode requirement covers. Apple Intelligence on the Mac runs on Apple silicon machines only.

The code is under the Apache 2.0 license and the package metadata on PyPI classifies it as alpha. Apple pushed 0.1.0 to PyPI on March 8, 2026, added the Attachment API for sending images alongside text in 0.2.0 on June 8, 2026 for WWDC 2026, and exposed the model’s context size and the token count of a given input in 0.2.1 on June 29, 2026. The README states the project is not yet taking contributions.

Alibaba’s Tongyi Lab published Mobile-Agent-v3.5 on February 15, 2026, describing GUI-Owl-1.5, a family of agent models that read a screenshot and emit the next tap, type or scroll. The family spans 2B, 4B, 8B, 32B and 235B-A22B parameters in instruct and reasoning variants, all built on Qwen3-VL. Every benchmark in the paper was run server-side against cloud sandboxes and emulators, and the authors give no on-device deployment, latency or hardware figures anywhere, so the small sizes are the only thing that points at a phone. The authors write that the smaller instruct variants, which skip the reasoning step, are fast enough to be deployed on edge devices and to work alongside a larger cloud model.

Across platforms, the authors report their 32B-Instruct model reaching 56.5 on OSWorld-Verified for desktop tasks, 47.6 on OSWorld-MCP and 46.8 on Mobile-World, the last two of which also require calling external tools. On AndroidWorld, which scores an agent driving real Android apps in an emulator, their 8B reasoning model reaches 71.6, which the paper puts just behind UI-TARS-2 at 73.3. For browsers, the paper’s table has the 32B reasoning model at 48.4 on WebArena, 46.6 on VisualWebArena and 82.1 on WebVoyager.

The small checkpoints carry a large part of that. The authors report 2B-Instruct at 43.5 on OSWorld-Verified and 67.9 on AndroidWorld, and 4B-Instruct at 48.2 and 69.8, which puts the 4B model within 2 points of their own 32B on Android tasks. On ScreenSpot-Pro, which tests whether a model can point at the right element in a high-resolution screenshot, they report 72.9 for 32B-Instruct and 80.3 when the model first crops a region and looks again. The paper’s prose and its tables disagree on which variant owns several headline numbers, and the figures above follow the tables.

Six checkpoints are on Hugging Face under the MIT license, covering 2B, 4B and 8B instruct, 8B reasoning, and 32B in both variants. The 235B-A22B model named in the paper is not among them. The code is MIT as well, and the repository points at a hosted cloud-sandbox demo rather than a local runtime.

Ali Chherawalla has posted Off Grid to Show HN, an Android and iOS app that runs text generation, image generation, vision and speech recognition entirely on the device. The submission reached 124 points and 66 comments. The repository is licensed under MIT.

The app bundles llama.cpp for text, Stable Diffusion for images and whisper.cpp for speech to text. It accepts any GGUF file and lists Qwen 3, Llama 3.2, Gemma 3 and Phi-4 among supported models; vision runs on SmolVLM, Qwen3-VL or Gemma 3n. It also connects to OpenAI-compatible servers on a local network, and builds a project knowledge base from PDFs chunked and embedded on-device with a bundled MiniLM model into SQLite.

The README reports 15 to 30 tokens per second on flagship CPUs and 20 to 40 tokens per second through Adreno GPUs via OpenCL on a Snapdragon 8 Gen 2 or newer. Image generation is given as 5 to 10 seconds on the Hexagon NPU and vision inference at about 7 seconds. The NPU path is marked experimental: it accelerates only Q4_0 and Q8_0 quantisation, and K-quants fall back to the CPU.

Comments disputed whether models of that size are usable. One reader wrote:

Ive tried a lot of self hosted models at a lot of sizes; those small models are not good enough, and do not have a context long enough to be useful for most everyday operations.

Chherawalla replied that “you don’t need claude level performance for a lot of day to day tasks” and that Qwen3 8B, or 4B quantised, is “actually quite good”.

Other threads compared the app with PocketPal AI, SmolChat and Google’s AI Edge Gallery, asked for an F-Droid build, and reported that image generation required network access on GrapheneOS. A reader on a Samsung device reported the interface did not move for the on-screen keyboard; the developer shipped a fix the same day.

Nvidia published Nemotron-Flash on November 24, 2025, a family of small language models built around measured decoding latency instead of parameter count. Nvidia reports Nemotron-Flash-1B decoding 8k tokens in 14.45 seconds on a single Nvidia H100 at batch size 1, against 27.55 seconds for Qwen3-0.6B, while averaging 49.63 across 16 benchmarks to Qwen3-0.6B’s 44.11. The paper was accepted at NeurIPS 2025.

The models mix three kinds of layer rather than using attention throughout. Nvidia interleaves DeltaNet and Mamba2 layers, operators that carry a fixed-size state instead of a cache that grows with the context, with a small number of ordinary attention layers, and an evolutionary search picked which operator goes where. Nemotron-Flash-1B holds 0.96B parameters in 12 blocks with only 2 full attention layers, and Nemotron-Flash-3B holds 2.7B parameters in 18 blocks with 3. Nvidia built both wider and shallower than other models of the same size, where Qwen3-0.6B stacks 28 layers, and writes that deep and thin models are parameter efficient but sit off the accuracy-latency frontier.

Nemotron-Flash-3B averages 60.98 across the same 16 tasks, which cover MMLU, commonsense reasoning, math, coding and recall, against 58.96 for Qwen2.5-3B and 55.47 for Qwen3-1.7B. On the same H100, with all models running through TensorRT-LLM AutoDeploy kernels and CUDA Graph, Nvidia measured 28.71 seconds for the 3B against 49.40 for Qwen2.5-3B and 36.20 for Qwen3-1.7B, so 1.7 and 1.3 times lower latency. Throughput was measured separately, with a 32k-token input at the largest batch size that still fits in memory, and Nvidia reports 2939 tok/s for the 3B against 459 and 157 for the two Qwen models, and 7289 tok/s for the 1B against 160 for Qwen3-0.6B. Every figure here is Nvidia’s own, taken on data centre GPUs, and the paper gives no phone or embedded measurements, with the earlier depth and operator studies run on an Nvidia A100.

Nvidia trained both models on 4.5T tokens using 256 H100 GPUs and released three checkpoints on Hugging Face, Nemotron-Flash-1B, the 3B base model and an instruction-tuned Nemotron-Flash-3B-Instruct, which averages 52.84 across MMLU, GPQA, GSM8K and IFEval against 48.17 for Qwen2.5-1.5B. The model cards set the licence to Creative Commons Attribution-NonCommercial 4.0, so the weights are downloadable but not licensed for commercial use, and neither Apache nor an Nvidia open model licence applies. Running them needs the TensorRT-LLM AutoDeploy attention kernel alongside the Mamba2 and FlashLinearAttention implementations, since Nvidia states that vLLM and TensorRT-LLM do not support DeltaNet-style linear attention on their own.

Researchers at Stanford University and Together AI profiled more than 20 local models on 1M real queries and report that at least one model with 20B or fewer active parameters answered 88.7% of them correctly. The best single model in their set, GPT-OSS-120B, reached 71.4% on its own, averaged across the four query collections. Their 2025 efficiency figure comes from that model running on an Apple M4 Max.

The paper proposes intelligence per watt, mean task accuracy divided by the mean power the accelerator draws while answering, so a pairing that gets more answers right on the same power budget scores higher. The best local pairing each year moved from 7.92×10⁻⁴ accuracy per watt for Mixtral 8x7B on an NVIDIA Quadro RTX 6000 in 2023, to 1.80×10⁻³ for Llama 3.1 8B on an RTX 6000 Ada in 2024, to 4.18×10⁻³ for GPT-OSS-120B on an Apple M4 Max in 2025, which the authors put at a 5.3 times gain over two years. Query coverage for that single best pairing rose from 23.2% to 71.3% across the same three years. A companion metric divides accuracy by the energy a query consumes instead, so slow generation counts against a device.

The 1M queries come from four collections. The team took 500K prompts from WildChat, a month of real ChatGPT traffic, 500K from NaturalReasoning, 12K from MMLU Pro and 26.5K from SuperGPQA, all single-turn. WildChat has no ground truth, so GPT-4o judged whether a local model’s answer matched one written by Qwen3-235B, while the other three collections supplied reference answers. Every run used a batch size of 1, one query at a time.

The hardware the authors call local is desktop and workstation class rather than mobile. Their 2025 numbers ran on an Apple Mac Studio with an M4 Max, 128 GB of unified memory and 480 W of continuous power, and the 2023 and 2024 points used NVIDIA workstation cards. No phone, tablet or wearable was profiled. Running the same Qwen3 models, the M4 Max reached 1.40 times lower accuracy per watt than an NVIDIA B200 and 1.6 to 2.3 times lower accuracy per joule, a gap the authors read as headroom for local accelerator design.

For the savings case the team simulated 80.2M queries over 24 hours split between four small models on M4 Max machines and Qwen3-235B in the cloud. Perfect routing to the smallest capable model cut energy by 80.4% against sending everything to the frontier model, and a router that picks correctly 80% of the time cut it by 64.3%, with the perfect case described as a theoretical maximum. The authors note that coverage falls in technical domains such as architecture and engineering, that GPT-OSS-120B answered 88.9% of WildChat queries against 64.9% of NaturalReasoning ones, and that they release their profiling harness so the measurements can be repeated on new hardware.

Meta Reality Labs published the MobileLLM-Pro technical report on November 10, 2025, describing a 1.08B-parameter language model built for phones and wearables. The model handles a context window of 128,000 tokens, four times what Gemma 3 1B and Qwen 3 0.6B accept. Quantising weights and embeddings to 4 bits takes the file from 2.2 GB in bf16 down to 590 MB for the CPU build and 720 MB for the accelerator build, which cannot share embedding weights.

Meta exported the model through ExecuTorch and measured it on a Samsung Galaxy S25 CPU with the XNNPACK backend and on a Samsung Galaxy S24 Hexagon Tensor Processor. On a 2,000-token prompt the paper reports 8.9 s of prefill on the CPU against 2.0 s on the Hexagon unit, and decode speeds of 33.6 tok/s and 31.6 tok/s while generating 1,000 tokens. At 8,000 tokens prefill climbs to 63.5 s on the CPU and 9.8 s on the Hexagon unit, decode falls to 19.7 and 22.8 tok/s, and the key-value cache grows from 14 MB to 40 MB.

The architecture puts 30 layers of width 1280 behind 20 attention heads and 4 key-value heads, and shares one weight matrix between the input embedding and the output head, which the authors say saves 260M parameters, about a quarter of the model. Three out of every four layers use local attention, comparing each token only against the 512 most recent ones, with a global layer every fourth one and at both ends. Meta’s model card credits that interleaving with cutting the key-value cache at 8,000 tokens from 117 MB to 40 MB and with 1.8 times lower prefill latency than global attention in every layer. Meta trained the model by distilling the logits of Llama 4 Scout under a KL divergence loss across three pretraining phases on roughly 1.64T tokens of open data, the second of them a step the authors call implicit positional distillation, which stretches the context window to 128k by having the teacher pass on the ability rather than feeding the student long documents.

Meta puts the base model 5.7 and 7.9 points above Gemma 3 1B and Llama 3.2 1B on average across the benchmarks its model card lists, and the paper’s table has it at 76.24 on BoolQ against 63.20 and 62.51, at 52.62 on ARC-Challenge against 38.40 and 38.28, and at 100% needle-in-a-haystack retrieval against 96.80 for Llama 3.2 1B. The instruction-tuned version scores 59.8 on HumanEval against 41.5 and 37.8, 44.8 on MMLU against Gemma’s 29.9 and Llama’s 49.3, and 62.0 on IFEval against Gemma’s 80.2. In Meta’s own human evaluation over 100 prompts per task, raters preferred Gemma 3 1B for summarisation 51 to 47 and for rewriting 49 to 45, while MobileLLM-Pro led on recall 47 to 30.

Meta released four checkpoints on Hugging Face, the base model, the instruction-tuned one, and two quantisation-ready base checkpoints produced by quantisation-aware training with self-distillation from the full-precision model. One uses 4-bit group-wise weights at group size 32 with 8-bit dynamic activations and an 8-bit key-value cache for CPU backends, the other 4-bit channel-wise weights for the Apple Neural Engine and Qualcomm’s Hexagon processor, which handle group-wise quantisation poorly or not at all. Needle-in-a-haystack retrieval holds at 96.44% and 98.67% against 100% at full precision. The weights sit behind a request form and carry the FAIR Noncommercial Research License.

Researchers at Zhejiang University, Zhejiang University of Technology and Hangzhou Dianzi University published Confidant in the ACM MobiCom 2025 proceedings on November 3, 2025, a day before the conference opened in Hong Kong. The framework fine-tunes a language model across several off-the-shelf devices instead of one, and the authors fine-tuned Phi2-2.7B on the Alpaca instruction dataset in 40.1 hours using three of them. Their write-up of that run names a Redmi K50 phone, a Core i7-13650HX laptop and a MacBook Pro with an M1 Pro, while the paper’s abstract and conclusion describe the same result as three mobile devices. The same trio fine-tuned BERT-Base in 2.2 hours.

Confidant cuts a model into consecutive slices, gives each device one slice, and passes activations from one device to the next, so no device ever holds the whole model. Batches flow through that chain as a pipeline, with each device starting a new batch while the devices behind it are still working on earlier ones. The device that holds the training data acts as coordinator and recalculates where the cuts fall during training, using each participant’s measured compute time, free memory and bandwidth, and it discards any split whose slice would not fit in a device’s memory. Inside a device, a second scheduler profiles the CPU and the GPU on varying numbers of attention heads and gives each processor as many heads as it can finish in roughly the same time, which is what the authors credit for running up to 1.94 times faster than a capacity-aware pipeline baseline they implemented themselves, and up to 3.82 times faster than the best single device, both on GPT-2 Medium.

Splitting the model is what brings the footprint per device down. The paper’s memory table puts fine-tuning BERT-Base at a batch size of 8 at 5.5 GB on a single Redmi K50, against a per-device average of 3.17 GB across three devices and 2.43 GB across four. Larger models leave no single-device figure to compare with, because a Redmi K50 fits only 1 of Phi2-2.7B’s 32 transformer blocks at a batch size of 4, and the per-device averages of 15.31 GB for Phi2-2.7B and 32.07 GB for LLaMA3-8B across three devices are carried by the two laptops rather than the phone. A separate table puts the memory PyTorch needs to fine-tune Phi2-2.7B at a batch size of 8 at 46.5 GB, next to the 8 GB to 12 GB of the four phones it lists.

The paper’s prose says Confidant almost does not cause any accuracy drop, while its own model performance table puts the three-device runs at a test loss of 0.66 on Phi2-2.7B and 3.55 on LLaMA3-8B, against 0.52 and 2.86 for fine-tuning the same models the conventional way on a single machine, where a lower loss is better. The four-device runs land at 0.76 and 2.85 on that same comparison, and BERT-Base token classification accuracy holds at 97.66% against 97.70%.

Keeping data on the device is the stated motivation. The authors write that training with Confidant does not require sharing raw data and that devices exchange only intermediate outputs and model gradients, which they present as the privacy argument for their two scenarios, a household assistant fine-tuned on family data during idle hours and a conference assistant fine-tuned on attendees’ phones. They also state that fine-tuning on a single cloud server would be faster and more straightforward, and that their aim is to use idle mobile hardware instead. The Android side is built on Alibaba’s MNN framework with laptops running PyTorch, about 10,900 lines of code in total, the paper is published under a Creative Commons Attribution 4.0 license, and the authors wrote that they planned to release the code in late 2025.

The PyTorch team released ExecuTorch 1.0 on October 22, 2025, the generally available version of its runtime for PyTorch models on phones, laptops and embedded devices. It takes a model straight from PyTorch without converting it to ONNX or TFLite and without rewriting it in another language, which the team says removes a common source of numerical mismatches and lost debug information. The beta arrived in October 2024.

Five backends moved from beta to production status. XNNPACK with Arm Kleidi covers CPU, Apple Core ML covers Apple silicon, the Qualcomm AI Engine delegate covers the Hexagon NPU, and Arm Ethos-U and Vulkan cover NPU and GPU. New backends arrived for Arm VGF, the NXP eIQ Neutron NPU, Samsung Exynos NPU and GPU, and Intel OpenVINO.

Meta says features built on ExecuTorch already run on Instagram, WhatsApp, Messenger and Facebook. Anuj Kumar, Director at Reality Labs, describes what it drives in Meta’s wearables, including the Ray-Ban Display glasses with EMG band.

Advanced AI features like speech recognition, motion sensing, and computer vision run on these devices by leveraging cutting-edge hardware.

Lysandre Debut, Chief Open-Source Officer at Hugging Face, says more than 80 percent of the most downloaded edge-friendly language models on Hugging Face run on ExecuTorch out of the box. The team validated coverage across object detection, depth, OCR, speech recognition and segmentation as well as text and multimodal models such as Voxtral for audio and Gemma 3 for image and text input. The main branch also allows embedding ExecuTorch into native C++ desktop applications.

Researchers at Georgia State University and Toyota InfoTech Labs published lm-Meter, a latency profiler that runs inside the inference engine on the phone and splits each generation into embedding, prefill, decode, softmax and sampling. Measuring the Pythia models on a Google Pixel 8 Pro, they report that scaling from 70M to 1.4B parameters raises prefill latency from 0.012 s to 1.9 s per input token, a 158x slowdown, while decode latency per output token grows from 0.015 s to 0.15 s, a 10x slowdown. The authors write that this inverts the server picture, where decode is usually the limiting phase for single-request inference.

Below the phase level the profiler times individual GPU kernels through OpenCL event timestamps, which give queue, submit, start and end times without access to the closed-source driver. Running a 4-bit quantised Gemma-2-2B-it on a Pixel 8 Pro, the authors report that fused matrix-multiplication kernels dominate a decode step and that the GPU sits idle for more than 21% of it, the second-largest contributor to the step, which they attribute to host-side data preparation and I/O stalls. The paged attention kernel that scans the growing key-value cache is the only one whose cost rises with position in the sequence, climbing from roughly 0.2 ms to about 0.8 ms per token over 250 decode steps, and idle time drops from about 21% to 12% when the model generates 256 tokens instead of 16.

Whether those measurements mean anything depends on what the profiler itself costs. lm-Meter sits in the MLC LLM runtime and TVM in about 3,500 lines of code and needs no host machine attached, and under the Powersave CPU governor, the most constrained setting they tested, the authors measure a throughput loss of 2.58% in prefill and 0.99% in decode. They put the same figures for MELTing Point, the on-device profiler they compare against, at 22% for prefill and more than 93% for decode. Checked against traces from Android GPU Inspector, they report end-to-end phase accuracy of at least 99.99% and mean kernel-level accuracy of 96.82% on the Pixel 8 Pro and 96.61% on a Pixel 7.

The code is on GitHub under the MIT license, with the MLC LLM path released for Android GPUs through OpenCL and support for llama.cpp, vLLM, iOS Metal and Nvidia Jetson listed as unfinished. The work was accepted to the ACM/IEEE Symposium on Edge Computing 2025 and funded by Toyota Motor North America. The measurements come from three phones, the Pixel 8 Pro, Pixel 7 and Pixel 6, and the authors state that other edge platforms such as Jetson boards and Intel NPUs may show different bottlenecks.

Meta released MobileLLM-R1 on September 12, 2025, a family of reasoning models at 140M, 360M and 950M parameters that write out a chain of intermediate steps before giving an answer instead of replying directly. Meta reports the 950M model scoring 74.0 on the MATH500 maths benchmark and 19.9 on LiveCodeBench v6, against 73.0 and 14.9 for Qwen3-0.6B. The technical report followed on September 29, 2025.

The three models saw 4.2T training tokens, which Meta puts at 11.7% of the 36T tokens behind Qwen3-0.6B. Pre-training ran in two phases of 2T tokens each, resampled from a curated pool of roughly 2T tokens of open datasets, followed by two mid-training phases of 100B tokens that distil from Llama-3.1-8B-Instruct as a teacher, then supervised fine-tuning on reasoning traces. Meta uses the result to argue that reasoning does not require the corpora above 10T tokens that recent models have been trained on.

For the post-trained models Meta’s table puts MobileLLM-R1-950M at 74.0 on MATH500, 67.5 on GSM8K, 15.5 on AIME 2024 and 19.9 on LiveCodeBench v6, where Qwen3-0.6B reaches 73.0, 79.2, 11.3 and 14.9 and DeepSeek-R1-Distill-Qwen-1.5B reaches 83.2, 77.3, 29.1 and 19.9. Among the models whose weights, data and recipe are all public, the same AIME figure compares with 0.6 for OLMo-2 1.48B and 0.3 for SmolLM2-1.7B. On code the 950M base model scores 46.3 on HumanEval against 30.5 for Qwen3-0.6B-Base, and the 140M base model reaches 15.9 where SmolLM2-135M scores 0.0.

Meta profiled the models through ExecuTorch on a Samsung Galaxy S22 with 8 GB of RAM, quantised to 4-bit weights with 8-bit dynamic activations. The 140M model generated 129.67 tok/s at a 1k context and 79.71 tok/s at 32k, the 950M model 31.05 tok/s at 1k before running out of memory at 16k, and Llama-3.2-1B 28.71 tok/s at 1k with the same memory failure. The architecture follows Meta’s earlier MobileLLM work and Llama 3.2, sharing one weight matrix between the input embedding and the output layer, and the post-trained models take a 32k context.

Meta released the weights on Hugging Face together with the training code, the data sources and the data mixing ratios on GitHub, under the FAIR Noncommercial Research License, which covers research, education and analysis but not commercial use. Access to the model repositories is gated behind a form.

Update, November 24, 2025. Meta released MobileLLM-R1.5 at the same three sizes, fine-tuned from the R1 models with one further round of on-policy distillation, where the student answers first and the teacher, here Llama-3.1-Nemotron-Nano-4B-v1.1, grades its own words back to it token by token. Meta reports the 950M version at 86.6 on MATH500, 82.6 on GSM8K, 39.9 on AIME 2024 and 29.1 on LiveCodeBench v6, up from 74.0, 67.5, 15.5 and 19.9 for MobileLLM-R1-950M, and says it beats DeepSeek-R1-Distill-Qwen-1.5B on every maths and coding benchmark it ran. At 360M Meta puts MATH500 at 63.4 against 28.4 and GSM8K at 52.8 against 24.5.

Apple announced the A19 Pro alongside the iPhone 17 Pro on September 9, 2025. Its 6-core GPU carries Neural Accelerators built into each core, which Apple says work together with the 16-core Neural Engine to power AI models, graphics and games. The company states that the chip enables running large local language models on the phone.

The A19 Pro also has a 6-core CPU that Apple calls the fastest in any smartphone, along with a larger cache and more memory than the A18 Pro. Combined with a vapor chamber cooling system, Apple puts sustained performance at up to 40 percent above the previous generation.

Researchers from Peking University and BUPT report up to 4.5 times faster end-to-end inference on a Xiaomi 14 with a Snapdragon 8 Gen 3, and 3.5 times faster on average for the attention kernel alone, with a peak of 6.9 times. The baseline is the arrangement most on-device frameworks use today, float32 full attention running on the CPU or GPU while the rest of the model sits on the NPU. The authors measured Qwen2-0.5B, Qwen2-1.5B, PhoneLM-0.5B and PhoneLM-1.5B on summarisation and two mobile agent tasks, and gave their own system only one of the phone’s middle CPU cores. Against the native attention of llm.npu, which uses four CPU cores or the GPU, they report up to 3.0 times lower latency from that one core. The paper is due at MobiSys 2026.

The starting problem is that attention keeps falling back off the NPU. Attention multiplies three activation tensors together, and mobile NPUs compile a static graph that fixes one quantisation scale factor for a whole tensor, so the authors measure an average 18 percentage point accuracy drop when the operator runs in INT8 on the NPU. Their trade is to stop asking the NPU for the answer and ask it only for the ranking. The NPU computes the query-key scores in INT8 purely to find which tokens matter, which the authors report recovers over 99 percent of the positions a float run would pick, since picking the largest values needs relative magnitudes rather than exact ones. The CPU then computes real attention in float over roughly 20 percent of the tokens and throws the other 80 percent away, at an average accuracy cost of 0.4 percentage points against the float baseline. Each attention head gets its own sparsity ratio, fixed by an offline profiling run the paper puts at about five minutes on a single A100.

On a Redmi K60 Champion Edition with a Snapdragon 8 Gen 2 the authors read battery voltage and current from the kernel and report a single attention kernel at 1024 tokens dropping from 8.59 J to 1.12 J for PhoneLM-1.5B, a factor of 7.66. Speedups on that older phone are smaller, 2 times, 1.25 times and 1.22 times across the three datasets, which the authors attribute to its Hexagon V73 NPU against the V75 in the Xiaomi 14. The paper reports no memory footprint figures. It does report that end-to-end inference time on the Xiaomi 14 stayed between 14.5 and 15.5 seconds while the phone was also taking photos, playing video, playing music or running a shopping app, since the LLM work is not competing for the cores those apps use.

The authors state a prototype of more than 10,000 lines of C++ and Python, built on Qualcomm’s QNN and Hexagon SDK for the NPU side and integrated into llm.npu, which they call the only fully open-source framework running on NPUs. The paper names no repository and announces no artefact release, and the title calls the system ShadowNPU while the text throughout calls the module shadowAttn. The arXiv listing carries a CC BY 4.0 license. For a hands-on point of comparison on the same silicon family, a user test of llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone put generation at about the same speed as the chip’s CPU, without the heat.