https://llmobile.kavents.com/tags/npu/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/llama-cpp-hexagon-npu-snapdragon/https://llmobile.kavents.com/ticker/llama-cpp-hexagon-npu-snapdragon/Thu, 10 Sep 2026 11:00:00 +0200A user has reported results from llama.cpp’s Hexagon NPU backend on a OnePlus 12 with a Snapdragon 8 Gen 3. They cross-compiled llama.cpp on Ubuntu following the project’s Snapdragon documentation and ran it from Termux on the phone. The reported figures are 8 tokens per second of prompt processing and 4.5 of generation with gemma-3-12b-it-qat at Q4_0, and 20 tokens per second of prompt processing and 12.5 of generation with gemma-3-4b-it-qat at Q4_0. According to the report the speed is about the same as running on the same chip’s CPU, but the phone does not get hot. The report lists several limitations. The backend accepts only Q4_0, IQ4_NL, MXFP4, Q8_0 and F32 files, and does not support KV cache quantisation. It runs on chips as old as the Snapdragon 888, but the poster notes that only the 8 Gen 2 and newer carry the tensor module used for language models. The Hexagon NPU can address 4 GB, so a model plus KV cache beyond that has to be split across more than one NPU device: LD_LIBRARY_PATH=./lib:/vendor/lib64 \ ADSP_LIBRARY_PATH=./lib \ GGML_HEXAGON_NDEV=2 \ ./bin/llama-completion \ -m /sdcard/gguf/gemma-3-12b-it-qat-Q4_0.gguf \ -ngl 99 --device HTP0,HTP1 \ -sys 'You are a helpful AI assistant' For context the poster cites 34 INT8 TOPS and 76.8 GB/s of memory bandwidth for the 8 Gen 3 NPU, against 80 TOPS and 228 GB/s for Qualcomm’s X2 Elite Extreme and 248 TOPS and 936 GB/s for an Nvidia RTX 3090. They ask readers with a Snapdragon 8 Elite Gen 5 phone or an X2 Elite Extreme laptop to post their own numbers, since those parts are said to address more than 4 GB and would not need multiple devices. Source: https://www.reddit.com/r/LocalLLaMA/comments/1t0k6fj/running_llamacpp_on_snapdragon_hexagon_npu_seems/ Read the article: https://llmobile.kavents.com/ticker/llama-cpp-hexagon-npu-snapdragon/QualcommAndroidNPUllama.cppGemmahttps://llmobile.kavents.com/ticker/iphone-18-pro-a20-pro-neural-engine/https://llmobile.kavents.com/ticker/iphone-18-pro-a20-pro-neural-engine/Wed, 09 Sep 2026 19:00:00 +0200Apple has introduced the iPhone 18 Pro and iPhone 18 Pro Max with the A20 Pro chip. According to Apple, the chip has a new dual 16-core Neural Engine, 32 cores in total, which the company describes as double the AI processing power of A19 Pro. Apple says the Neural Engine accelerates on-device AI models and computational photography. Image: Apple. The A20 Pro also offers 50 percent more memory bandwidth than A19 Pro, and its 6-core CPU includes integrated Neural Accelerators. Apple positions the chip for “more advanced on-device AI workloads” alongside games. The phones ship with iOS 27, Apple Intelligence and the new Siri AI. Apple states that Apple Intelligence uses on-device processing together with Private Cloud Compute. Pre-orders start on 12 September, with availability from 18 September; prices start at $1,199 for the iPhone 18 Pro and $1,299 for the iPhone 18 Pro Max, both with 256 GB. Source: https://www.apple.com/newsroom/2026/09/apple-debuts-iphone-18-pro-and-iphone-18-pro-max/ Read the article: https://llmobile.kavents.com/ticker/iphone-18-pro-a20-pro-neural-engine/AppleiPhoneChipsNPUhttps://llmobile.kavents.com/ticker/pixel-11-tensor-g6/https://llmobile.kavents.com/ticker/pixel-11-tensor-g6/Wed, 12 Aug 2026 19:00:00 +0200Google has announced the Pixel 11, Pixel 11 Pro and Pixel 11 Pro XL, built around the Tensor G6 chip. Google states that Tensor G6 packs 50 percent more TPU compute and, paired with the latest Gemini Nano model, processes on-device AI tasks up to 3.5 times faster while using up to 3.5 times less energy. The company also cites an upgraded CPU with 25 percent faster web browsing and 15 percent quicker app launches, and says the chip powers the 30x Super Zoom on the 5x telephoto lens. Google does not publish RAM figures, model sizes or per-task latency in the announcement. Pre-orders opened on 12 August, with retail availability from 20 August. ▶Meet Google Pixel 11 ProLoading connects your browser to www.youtube-nocookie.com, which may process your IP address and use cookies. Load contentOpen externallyVideo: Made by Google. Source: https://blog.google/products-and-platforms/devices/pixel/google-pixel-11-pro-xl/ Read the article: https://llmobile.kavents.com/ticker/pixel-11-tensor-g6/GooglePixelChipsNPUGemini Nanohttps://llmobile.kavents.com/ticker/mlperf-inference-mobile/https://llmobile.kavents.com/ticker/mlperf-inference-mobile/Mon, 15 Jun 2026 17:00:00 +0200MLCommons announced MLPerf Mobile v6.0 on June 15, 2026, adding generative AI tests that run Llama on an Android device and report token throughput. The new workload comes in 1B, 3B and 8B parameter sizes, using Llama 3.2 1B Instruct, Llama 3.2 3B Instruct and Llama 3.1 8B Instruct. Each size runs twice, once against TinyMMLU, a small multiple-choice knowledge set, and once against IFEval, which checks whether a model obeys the instructions in a prompt, according to the task configuration in the app repository. MLCommons says the tests run on devices with enough memory through the CPU without tailored acceleration, and that v6.0 adds NPU-accelerated execution of Llama 3.1 8B Instruct on Qualcomm Snapdragon 8 Elite Gen 5. The v6.0 app release of June 9, 2026 also brings support for MediaTek Dimensity 9500 chips and updates support for the Samsung Exynos 2600. The rest of the suite covers image classification with MobileNetV4 on ImageNet, object detection on COCO 2017, segmentation with MOSAIC on ADE20K, language understanding with MobileBERT on SQuAD 1.1, super resolution with EDSR, and Stable Diffusion 1.5 on COCO captions, as the benchmark page lists them. Almost every test runs in the single stream scenario, where the score is the 90th percentile latency for one query at a time, and image classification also runs an offline pass scored as throughput. Stable Diffusion, the first generative test in the suite, arrived in the v4.1 app release of February 11, 2025. Mostafa El-Khamy, co-chair of the MLPerf Mobile working group, had set out the plan in May 2024. Looking ahead, the MLPerf Mobile group is working to add generative AI tasks to a future version of the benchmark suite. Results come from chip and device makers that run the app and submit their logs. The published v6.0 results hold one submission in the closed division, from Samsung on an Exynos 2600 phone, and it covers the vision tests, language understanding, super resolution and Stable Diffusion rather than the new Llama ones. Samsung’s logs give a 90th percentile single stream latency of 0.56 ms for image classification, 0.83 ms for MobileBERT and 1.87 s for one Stable Diffusion image. The v5.0 round carried submissions from Samsung and Qualcomm Innovation Center. Screenshots: Samsung's submission to MLPerf Mobile v6.0, published by MLCommons. Submissions are held back until review closes under the MLCommons submission rules, which keep every result confidential inside a review committee after the deadline and let submitters file objections that the committee then decides. MLCommons publishes the accepted runs in the results table on the MLPerf Inference Mobile page, with the logs and calibration notes in a per-round repository on GitHub. The app is available through Google Play, the Apple App Store and the MLPerf Mobile repository under the Apache 2.0 license. Source: https://mlcommons.org/2026/06/mlperf-mobile-v6/ Read the article: https://llmobile.kavents.com/ticker/mlperf-inference-mobile/BenchmarksAndroidLlamaNPUQualcommhttps://llmobile.kavents.com/ticker/on-device-diffusion-llm-npu/https://llmobile.kavents.com/ticker/on-device-diffusion-llm-npu/Thu, 11 Jun 2026 14:44:00 +0200Tuowei Wang, Yanfan Sun and Ju Ren of Tsinghua University and Beihang University published llada.cpp on June 11, 2026, an inference framework that runs diffusion language models on a phone’s NPU. On an OnePlus Ace5 Pro with a Snapdragon 8 Elite, the authors measure LLaDA-8B-Instruct generating 128 tokens 38 times faster than the same model on the phone’s CPU for GSM8K prompts and 42 times faster on BoolQ, and they report a range of 17 times to 42 times across their phones and tasks. The same figure puts Llama-3-8B-Instruct on the NPU at 19 to 21 times that CPU baseline, and the authors state that llada.cpp runs LLaDA-8B up to 3.9 times faster than an autoregressive model of the same size. Chart: Wang, Sun and Ren, CC BY 4.0. A diffusion language model does not emit one token after another. It starts from a block of masked positions, 32 tokens by default here, and refines all of them together over repeated denoising steps, committing the positions it is most confident about and carrying the rest into the next step. That turns decoding into a run of large matrix operations, which is the work a mobile NPU is built for, while autoregressive decoding hands the NPU one token’s worth of work per step. The cost the paper identifies is that every denoising step recomputes the full sequence, so the parallel work gets repeated many times over. llada.cpp is built on llama.cpp and targets Qualcomm’s Hexagon NPU with 4-bit weights. Once a block is nearly done only a few masked tokens are left, so the framework pulls tokens from the next block into the same NPU pass to keep it full, keeps already committed tokens open to revision and repairs the unstable ones on the CPU while the NPU carries on, and repacks the limited address space the NPU can see so weights and cache do not have to be remapped between steps. The authors’ breakdown of a 128-token GSM8K request on the Snapdragon 8 Elite puts it at 2996.2 s on the CPU, 607.0 s once the prefix KV cache is reused, and 16.1 s with the full framework. Accuracy moves in both directions. Measured on 200-sample subsets against LLaDA-8B-Instruct on the CPU, the authors’ table has GSM8K rising from 39.0 to 43.5 and ARC-C from 84.0 to 85.0, while BoolQ falls from 82.5 to 80.5 and HellaSwag from 51.0 to 49.5, and their ablation shows the borrowed future-block tokens costing accuracy on all four tasks before the CPU-side revision path recovers it. The work covers three 16 GB phones, an OnePlus 12 with a Snapdragon 8 Gen 3, the Ace5 Pro, and an OnePlus 15 with a Snapdragon 8 Elite Gen 5, and the authors state that on the Snapdragon 8 Gen 3 their framework is still slower than Llama-3-8B at 128 tokens and only pulls ahead on the two newer chips. The paper is on arXiv under CC BY 4.0 and names no code release. Source: https://arxiv.org/abs/2606.13740 Read the article: https://llmobile.kavents.com/ticker/on-device-diffusion-llm-npu/QualcommNPUAndroidllama.cppResearchhttps://llmobile.kavents.com/ticker/quant-npu/https://llmobile.kavents.com/ticker/quant-npu/Tue, 19 May 2026 12:48:00 +0200Eight authors published Quant.npu on May 19, 2026, a way of quantising 1B to 3B language models so they run on a phone’s neural processing unit with no rescaling at runtime. On a Qualcomm SM8650 NPU they measure their 4-bit weight, 8-bit activation build of Llama-3.2-3B-Instruct finishing up to 15.1 percent faster than the same model under ExecuTorch’s 4-bit per-block weights with 16-bit activations, for about 1 percentage point of zero-shot accuracy. Across every model and task in the paper the authors put the average cost at a 2.58 percent accuracy drop and a 1.23 rise in perplexity. Quantisation stores a tensor as small integers plus a scale factor that maps them back to real values. Most published 4-bit methods, including the QuaRot and SpinQuant work Quant.npu is measured against, work that scale out while the model runs by scanning each activation tensor for its largest value. A mobile NPU compiles its graph before the model ships and expects every scale to be fixed by then, because that scan is a reduction the hardware is poor at, at least 7 vector instruction cycles for a 128-element 8-bit vector by the authors’ count, and because two of the four rotation steps these methods use against outliers need floating-point matrix multiplications during inference, which the paper puts at roughly double the compute. Anything left dynamic therefore has to run outside the compiled integer pipeline, and the authors report that simply freezing a model optimised under dynamic quantisation costs 15.61 percent accuracy on SmolLM2-1.7B-Instruct. Quant.npu learns the scales during optimisation instead and folds its two rotation matrices into the weights offline, so the shipped graph is integer arithmetic end to end. In the paper’s main table for Llama-3.2-3B-Instruct at 4-bit weights and 8-bit activations, that lifts average zero-shot accuracy across six benchmarks to 58.27 percent against 46.23 for SpinQuant and 47.66 for QuaRot in the same static setting, with perplexity on C4 falling from 28.78 to 19.16. Against the slower ExecuTorch configuration it beats on latency, though, Quant.npu is the one behind, scoring lower on five of the six benchmarks, 1.08 points lower on average and 0.97 higher in perplexity, and FP32 stays ahead of both at 61.3 percent. On the newer SM8750 NPU the authors report peak memory of 1876 MB to 1985 MB for their 3B builds against 2216 MB to 2435 MB for ExecuTorch, and prefill of 969.72 tok/s for Qwen2.5-3B-Instruct on the DroidTask set against 808.10. They name two limits of their own, that the pipeline still keeps a few operations such as the SiLU activation at 16 bits to hold accuracy, which they call a bottleneck against a fully low-bit pipeline, and that they use no method for picking calibration data even though the optimisation is sensitive to it. Quantising Qwen3-1.7B takes about 2.5 hours on NVIDIA A40 GPUs, and the preprint announces no code release. Source: https://arxiv.org/abs/2605.20295 Read the article: https://llmobile.kavents.com/ticker/quant-npu/QualcommNPUQuantisationLlamaResearchhttps://llmobile.kavents.com/ticker/llm-inference-at-the-edge/https://llmobile.kavents.com/ticker/llm-inference-at-the-edge/Tue, 24 Mar 2026 19:28:00 +0100Four authors at Conscious Engines published a benchmark of Qwen 2.5 1.5B at 4-bit across four edge platforms on March 24, 2026, feeding each device the same 258-token prompt 20 times in a row with a 1-second gap instead of once. On an iPhone 16 Pro the first two runs peak at 40.49 tokens per second, and from run 17 onwards the phone holds a plateau of 23.67 tok/s, which the authors put at 41.5 percent below that peak. The iPhone passes through the three thermal states iOS reports to an app. Runs 1 and 2 sit in Normal and average 36.83 tok/s, runs 3 to 16 sit in Warm and average 24.96 tok/s, and runs 17 to 20 sit in Hot at 23.67 tok/s with a variation of 1.2 percent. The authors write that the 1-second gap allows no thermal recovery, and they measure 5 percent of battery drained over the 20 runs, which they project to roughly 400 replies of this length per charge. A Samsung Galaxy S24 Ultra degrades more gently in the same test. Its Adreno 750 GPU boosts to 1,000 MHz on runs 4 and 5 for a peak of 12.21 tok/s, then steps down to a 720 to 770 MHz plateau from run 8 where the authors measure 10.38 tok/s, 15 percent off the peak, with a low of 9.55 tok/s on the final run. They stress that reaching this plateau took deliberate tuning, with the display off, the context capped at 2,048 tokens and MLC-LLM’s prefill chunk cut to 128 tokens, and that earlier unconstrained runs hit an OS-enforced GPU frequency floor. Neither phone ran the model on its NPU. The paper states that MLX dispatches to the A18 Pro GPU through Metal and does not target Apple’s Neural Engine, and that the S24 Ultra ran through MLC-LLM on the Adreno GPU, leaving the Snapdragon 8 Gen 3’s Hexagon NPU outside the measurement. The one NPU that did run the model, a Hailo-10H module in a Raspberry Pi 5, was also the only part that never throttled, holding 6.914 tok/s across all 20 runs with a variation of 0.04 percent at under 2 W, its CPU at 52.7°C and the NPU at 58.5°C, with attention layers that hailo-ollama does not offload falling back to the Pi’s Cortex-A76 CPU. A user running llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone reported 12.5 tok/s of generation on Gemma 3 4B and said the phone stayed cool. A laptop RTX 4050 on battery led on speed at 131.7 tok/s and 34.12 W, its GPU climbing from 55°C to 70°C without throttling. Energy per token comes out at 270.5 mJ for the Hailo module measured across the whole system, 297.3 mJ for the RTX 4050 measured at the GPU, and 143.0 mJ for the S24 Ultra plateau derived from the Android fuel gauge with the display off. The authors state that the three readings differ in scope and call the resemblance suggestive rather than settled, and report no energy figure for the iPhone because iOS exposes no per-component power reading. They tested one model, one prompt and one unit per platform across four different runtimes, and describe the results as characterising hardware and software together rather than hardware on its own. Source: https://arxiv.org/abs/2603.23640 Read the article: https://llmobile.kavents.com/ticker/llm-inference-at-the-edge/BenchmarksNPUiPhoneSamsungResearchhttps://llmobile.kavents.com/ticker/flexserve/https://llmobile.kavents.com/ticker/flexserve/Tue, 10 Mar 2026 01:31:25 +0100Researchers at Shanghai Jiao Tong University published FlexServe on March 10, 2026, a serving system that keeps an on-device language model’s weights and the text going through it inside ARM TrustZone’s secure world. Running Llama 3.2 3B, Llama 3.1 8B, Qwen3 1.7B and Qwen3 8B quantised to 8-bit on a NanoPC-T6 board with a Rockchip RK3588, 16 GB of memory and a 6 TOPS NPU, the authors measure time to first token 4.41 percent above the same inference run without any protection, averaged over their prompt lengths. Against a TrustZone baseline that already uses a secure NPU and a loading pipeline, they report FlexServe 1.43 to 2.42 times faster, 1.85 times on average. TrustZone splits the chip into a normal world, where Android and every app runs, and a secure world the normal world cannot read. FlexServe puts the inference engine, the weights, the prompt, the generated text and the KV cache on the secure side, so a kernel an attacker has taken over cannot read any of it out of memory, and the authors justify that assumption with Linux’s 40 million lines of code and 9756 CVEs. Both halves are worth stealing in their account, since a vendor’s fine-tuned weights cost real money to train yet sit on a stranger’s device, and the apps calling a shared model feed it chat history and screen contents. The authors name the boundary they cannot cross, which is that an app in the normal world still receives its own input and output there, so a compromised kernel can take those, and only the weights stay protected in that case. Plain TrustZone is slow at this because it hands out secure memory as a few large contiguous regions. Reserving 8 GB at runtime therefore means the kernel’s contiguous allocator compacting scattered pages, which the authors time at 6440.67 ms against 568.58 ms for FlexServe’s page-granular secure memory, and the secure world cannot normally touch the NPU at all, leaving 30.06 s of CPU computation where the NPU needs 1.94 s. FlexServe carves secure pages out with the stage-2 page tables that ARM’s virtualisation extension already provides, and flips the NPU between protected and unprotected mode in 0.21 ms. Handing 8 GB back drops from 732.85 ms to 80.50 ms, because the pages are released lazily rather than zeroed in one pass. A scheduler decides which models stay resident when several share the device, holding the earliest layers of each in secure memory so the pipeline does not stall and prefetching the next model while the current one is still generating. With a 4 GB cache and requests drawn at random from UltraChat, OpenAssistant, Dolly and Alpaca, the authors measure pairs of models answering 8.84 times faster than the plain secure-world baseline and 2.53 times faster than the optimised one. Across ten agent workflows, where one model’s finished answer feeds the next, the gains reach 24.30 times and 4.05 times, averaging 14.15 and 2.94. Serving one phone-side model to several callers is also what ElastiLM works on, by resizing the shared model to each request’s latency budget rather than isolating it. The protection costs the rest of the device something as well. A SQLite instance holding 3 GB in the normal world runs 1.31 times slower while FlexServe holds 8 GB in the background, against 3.27 times slower with the contiguous allocator, and the authors put the standing virtualisation cost at 2.46 percent on SPEC CPU, which they switch off when no secure inference is running. They list the prototype’s limits themselves, a development board rather than a phone, Llama 3 and Qwen3 series models only, no GPU in the pipeline, and an NPU runtime slow enough that computation becomes the bottleneck, so a faster NPU would widen their lead. Side-channel attacks, physical attacks and denial of service are out of scope, and the paper names no code release. Source: https://arxiv.org/abs/2603.09046 Read the article: https://llmobile.kavents.com/ticker/flexserve/ResearchArmNPUMemoryLlamahttps://llmobile.kavents.com/ticker/executorch-1-0/https://llmobile.kavents.com/ticker/executorch-1-0/Wed, 22 Oct 2025 18:00:00 +0200The PyTorch team released ExecuTorch 1.0 on October 22, 2025, the generally available version of its runtime for PyTorch models on phones, laptops and embedded devices. It takes a model straight from PyTorch without converting it to ONNX or TFLite and without rewriting it in another language, which the team says removes a common source of numerical mismatches and lost debug information. The beta arrived in October 2024. Diagram: PyTorch documentation. Five backends moved from beta to production status. XNNPACK with Arm Kleidi covers CPU, Apple Core ML covers Apple silicon, the Qualcomm AI Engine delegate covers the Hexagon NPU, and Arm Ethos-U and Vulkan cover NPU and GPU. New backends arrived for Arm VGF, the NXP eIQ Neutron NPU, Samsung Exynos NPU and GPU, and Intel OpenVINO. Meta says features built on ExecuTorch already run on Instagram, WhatsApp, Messenger and Facebook. Anuj Kumar, Director at Reality Labs, describes what it drives in Meta’s wearables, including the Ray-Ban Display glasses with EMG band. Advanced AI features like speech recognition, motion sensing, and computer vision run on these devices by leveraging cutting-edge hardware. Lysandre Debut, Chief Open-Source Officer at Hugging Face, says more than 80 percent of the most downloaded edge-friendly language models on Hugging Face run on ExecuTorch out of the box. The team validated coverage across object detection, depth, OCR, speech recognition and segmentation as well as text and multimodal models such as Voxtral for audio and Gemma 3 for image and text input. The main branch also allows embedding ExecuTorch into native C++ desktop applications. Source: https://pytorch.org/blog/introducing-executorch-1-0/ Read the article: https://llmobile.kavents.com/ticker/executorch-1-0/MetaPyTorchDeveloper toolsNPUOpen sourcehttps://llmobile.kavents.com/ticker/a19-pro-neural-accelerators/https://llmobile.kavents.com/ticker/a19-pro-neural-accelerators/Tue, 09 Sep 2025 20:00:00 +0200Apple announced the A19 Pro alongside the iPhone 17 Pro on September 9, 2025. Its 6-core GPU carries Neural Accelerators built into each core, which Apple says work together with the 16-core Neural Engine to power AI models, graphics and games. The company states that the chip enables running large local language models on the phone. The A19 Pro also has a 6-core CPU that Apple calls the fastest in any smartphone, along with a larger cache and more memory than the A18 Pro. Combined with a vapor chamber cooling system, Apple puts sustained performance at up to 40 percent above the previous generation. Image: Apple. The vapor chamber Apple credits for the sustained performance gain. Source: https://www.apple.com/newsroom/2025/09/apple-unveils-iphone-17-pro-and-iphone-17-pro-max/ Read the article: https://llmobile.kavents.com/ticker/a19-pro-neural-accelerators/AppleiPhoneChipsApple SiliconNPUhttps://llmobile.kavents.com/ticker/shadownpu/https://llmobile.kavents.com/ticker/shadownpu/Fri, 22 Aug 2025 09:41:00 +0200Researchers from Peking University and BUPT report up to 4.5 times faster end-to-end inference on a Xiaomi 14 with a Snapdragon 8 Gen 3, and 3.5 times faster on average for the attention kernel alone, with a peak of 6.9 times. The baseline is the arrangement most on-device frameworks use today, float32 full attention running on the CPU or GPU while the rest of the model sits on the NPU. The authors measured Qwen2-0.5B, Qwen2-1.5B, PhoneLM-0.5B and PhoneLM-1.5B on summarisation and two mobile agent tasks, and gave their own system only one of the phone’s middle CPU cores. Against the native attention of llm.npu, which uses four CPU cores or the GPU, they report up to 3.0 times lower latency from that one core. The paper is due at MobiSys 2026. The starting problem is that attention keeps falling back off the NPU. Attention multiplies three activation tensors together, and mobile NPUs compile a static graph that fixes one quantisation scale factor for a whole tensor, so the authors measure an average 18 percentage point accuracy drop when the operator runs in INT8 on the NPU. Their trade is to stop asking the NPU for the answer and ask it only for the ranking. The NPU computes the query-key scores in INT8 purely to find which tokens matter, which the authors report recovers over 99 percent of the positions a float run would pick, since picking the largest values needs relative magnitudes rather than exact ones. The CPU then computes real attention in float over roughly 20 percent of the tokens and throws the other 80 percent away, at an average accuracy cost of 0.4 percentage points against the float baseline. Each attention head gets its own sparsity ratio, fixed by an offline profiling run the paper puts at about five minutes on a single A100. On a Redmi K60 Champion Edition with a Snapdragon 8 Gen 2 the authors read battery voltage and current from the kernel and report a single attention kernel at 1024 tokens dropping from 8.59 J to 1.12 J for PhoneLM-1.5B, a factor of 7.66. Speedups on that older phone are smaller, 2 times, 1.25 times and 1.22 times across the three datasets, which the authors attribute to its Hexagon V73 NPU against the V75 in the Xiaomi 14. The paper reports no memory footprint figures. It does report that end-to-end inference time on the Xiaomi 14 stayed between 14.5 and 15.5 seconds while the phone was also taking photos, playing video, playing music or running a shopping app, since the LLM work is not competing for the cores those apps use. Diagram: Yin et al. The authors state a prototype of more than 10,000 lines of C++ and Python, built on Qualcomm’s QNN and Hexagon SDK for the NPU side and integrated into llm.npu, which they call the only fully open-source framework running on NPUs. The paper names no repository and announces no artefact release, and the title calls the system ShadowNPU while the text throughout calls the module shadowAttn. The arXiv listing carries a CC BY 4.0 license. For a hands-on point of comparison on the same silicon family, a user test of llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone put generation at about the same speed as the chip’s CPU, without the heat. Source: https://arxiv.org/abs/2508.16703 Read the article: https://llmobile.kavents.com/ticker/shadownpu/QualcommNPUResearchQwenAndroidhttps://llmobile.kavents.com/ticker/pd-device-cloud-split/https://llmobile.kavents.com/ticker/pd-device-cloud-split/Tue, 12 Aug 2025 17:56:29 +0200Thirty researchers at Huawei Technologies published P/D-Device on August 12, 2025, a scheme that runs a single language model request across a cloud data centre and a phone at the same time instead of choosing one of them. The cloud reads the prompt and writes only the first few tokens, then the device carries the answer to the end. The authors measure average time to first token falling by at least 60 percent against running the whole request on the device, and cloud throughput rising by 1.6 to 15 times, 7.6 times on average, against letting the cloud generate a full 200-token answer. The split follows from the two phases of inference wanting different hardware. Prefill, where the model reads the prompt before it writes anything, is compute-bound, and the authors report on-device time to first token climbing steeply with prompt length, into seconds or tens of seconds once prompts run to several thousand tokens. Decoding, where the model emits one token at a time, behaves differently, because a phone serves one user while a cloud instance serves a large batch, so the authors find the two roughly matched at tens of milliseconds per output token. Decoding is also what occupies a cloud instance for tens of seconds per request, so the scheme cuts the cloud off after a set number of tokens and hands the rest of the answer to the device. The device displays the cloud’s first token the moment it arrives, so the wait the user actually perceives is the cloud’s prefill, which the authors put at hundreds of milliseconds. Behind that display the device runs its own prefill while a speed controller releases the next tokens from the cloud at a deliberately slowed pace, matched to human reading speed, until the device has caught up and generates the rest itself. During its prefill the cloud also scores the prompt using the attention weights it has already computed, picks the sentences that matter most, and returns the selection as a compressed mask so the device prefills a shorter prompt, with a quarter of the prompt kept by default. What crosses the network is text, never model state. The device sends the prompt as raw text and the cloud returns the first token, the mask and a token budget, which the authors cap at hundreds of kilobytes for an 8k prompt, while the KVCache, the gigabyte-scale intermediate state that other cloud-device schemes ship back and forth, is too large to move in time and is never transferred at all. On the prompt text itself the authors state that the request goes out only under user and app authorisation and after sensitive information such as phone numbers and real names has been removed or substituted, and they cite Apple’s Private Cloud Compute as the class of guarantee they have in mind, without describing an implementation or measuring one. The archive’s post on Flower Intelligence covers the reverse arrangement, where the device runs first and the remote service stays off unless the app turns it on. Diagram: Jin et al. The cloud scores the prompt during its own prefill and returns the selection as a mask alongside the first token. The prototype runs cloud inference on Huawei Ascend NPUs and device inference on Kirin SoCs, tested on a phone and a tablet, with every chip and model anonymised as Type 1, Type 2, Model 1 and Model 2 throughout the paper. On LongBench with the same model on both sides, the authors’ own table gives their refinement an average of 26.64 on single-document QA against 25.89 for the unrefined on-device baseline, and 26.83 against 22.96 on multi-document QA. The authors name the costs themselves, a quality score around 85 percent of cloud-only inference when the cloud runs the larger model, refinement adding one or two hundred milliseconds at an 8k prompt and several hundred under load, and the smoothed display running slightly above their own tolerable threshold of about a hundred milliseconds per token once prompts pass 8k. The paper, on arXiv under a CC BY 4.0 license, further states that connections were tested over Wi-Fi with an IP whitelist rather than LTE, that collaboration across different NPU types is still being explored, and that the device supports prompts of several thousand tokens against hundreds of thousands in the cloud. Source: https://arxiv.org/abs/2508.09035 Read the article: https://llmobile.kavents.com/ticker/pd-device-cloud-split/ResearchHuaweiNPUBenchmarkshttps://llmobile.kavents.com/ticker/generative-ai-at-the-edge-queue/https://llmobile.kavents.com/ticker/generative-ai-at-the-edge-queue/Wed, 21 May 2025 18:00:00 +0200Vijay Janapa Reddi of Harvard University published an article in ACM Queue on May 21, 2025, arguing that generative models reach phones, glasses and robots only if the field stops chasing parameter counts and starts designing for deployment. He projects more than 50 billion edge devices by 2030 and writes that today’s frontier models, often hundreds of billions of parameters, cannot leave the data centre. His deployment matrix places only models under 1B parameters in the viable column for true edge hardware, puts medium models of 1B to 10B parameters within reach of high-end phones and single GPUs, and confines anything larger to servers or the cloud. Reddi grounds that in memory first. He writes that a 6B parameter model at half precision needs roughly 12 GB for weights alone, far above typical mobile RAM, that even a 1B model at about 2 GB can strain some devices, and that small models usually occupy 100 MB to 2 GB. He puts the latency bar at around 50 ms for an augmented reality translation to feel instantaneous and under a second for a dialogue assistant to feel fluid, and he notes that batching across users, which keeps cloud serving efficient, does not apply when requests arrive one at a time. On energy he cites an int8 quantised model using about five times less energy per query than an FP16 one on the same hardware, and says MLPerf Power results suggest model energy efficiency is plateauing even as hardware improves. Reddi frames the whole problem as a data-model-compute triangle, where scarce local data, limited compute and the need for compact models each worsen the others, and he argues compression alone will not close the gap, since smaller models hallucinate more and can lose their refusal behaviour when quantised hard. He proposes measuring capability and cost together through metrics such as hallucinations per watt-hour, running a model on factual question answering until it consumes 1 Wh and counting the wrong statements, and toxicity per token normalised by model size. The article ran in Queue volume 23, issue 2, and the full text is free to read on queue.acm.org. Source: https://dl.acm.org/doi/abs/10.1145/3733702 Read the article: https://llmobile.kavents.com/ticker/generative-ai-at-the-edge-queue/ResearchMemoryQuantisationNPUWearableshttps://llmobile.kavents.com/ticker/genai-at-the-edge-survey/https://llmobile.kavents.com/ticker/genai-at-the-edge-survey/Wed, 19 Feb 2025 21:43:00 +0100Seven researchers at Johns Hopkins University and Duke University published a survey of generative AI on edge devices on February 19, 2025. Most of the accelerator hardware it collects has not been built. Its table of 12 accelerators for language and diffusion models covers 8 simulator studies against 4 fabricated chips, and the highest peak energy efficiency in it is 74.34 TOPS/W, from a 28nm computing-in-memory chip designed for Stable Diffusion. Compression is the part the survey treats as settled enough to quantify, and it attributes every figure to the work it reviews. It reports unstructured pruning reaching up to 60 percent sparsity on large models and a sparsity-aware inference system turning that into a 3 times speedup, while structured pruning that removes whole components gives about 2 times with little quality loss. Quantisation has been taken to 1 bit per weight in work such as BitNet, and the survey names what is still open there as holding generation quality under extreme compression and finding efficient ways to train quantised models on the edge devices themselves rather than only to run them. The frameworks chapter is where the survey names hardware a reader can buy. It cites a measurement that applying NVIDIA’s TensorRT to transformer models on a Jetson Xavier cuts latency by over 60 percent with negligible accuracy loss, and points to MELTing Point, a benchmark suite that tracks language model energy use and memory footprint across smartphones and Jetson boards. The authors state that no dedicated survey of generative AI at the edge existed before theirs, close by naming model personalisation and security across distributed edge nodes as the challenges that persist, and publish on arXiv under a Creative Commons Attribution 4.0 license. Diagram: Navardi and colleagues, from the survey. Source: https://arxiv.org/abs/2502.15816 Read the article: https://llmobile.kavents.com/ticker/genai-at-the-edge-survey/ResearchChipsQuantisationNPUhttps://llmobile.kavents.com/ticker/bluelm-v/https://llmobile.kavents.com/ticker/bluelm-v/Sat, 16 Nov 2024 01:14:51 +0100Researchers at CUHK MMLab and vivo AI Lab published BlueLM-V-3B on November 16, 2024, a vision-language model designed together with the code that runs it on a phone. The authors deployed it on the NPU of a MediaTek Dimensity 9300 in a vivo X100 and report a peak memory requirement of 2.2 GB and a generation speed of 24.4 tok/s with the language model weights quantised to 4-bit. The model pairs a 2.7B in-house BlueLM language model with a 400M SigLIP vision encoder that takes 384 by 384 image tiles. The authors report an average score of 66.1 on OpenCompass, a suite of eight multimodal benchmarks, which they state is the highest among models of 4B parameters or fewer, ahead of the 8B models MiniCPM-V-2.6 and InternVL2-8B. Most of the speed comes from how the model decides what resolution to feed the encoder. Dynamic resolution schemes pick a grid of tiles to fit an image, and the authors show that existing choices enlarge pictures a lot, with LLaVA-NeXT resizing and padding a 394 by 390 image to 768 by 768, four times the area, and InternVL 1.5 stretching a 380 by 76 image to 1920 by 384, twenty-five times the area. More pixels mean more tiles and more image tokens, so BlueLM-V-3B uses what the authors call relaxed aspect ratio matching, which tolerates a slightly smaller effective resolution when it wastes less of the canvas. Both examples then land on a single 1:1 tile, and across the 665k-sample LLaVA training set the method picks a smaller grid than InternVL 1.5 in 523k cases. The rest is shaped around what an NPU is bad at. The authors encode image tiles in batches of four at a time, which they measured as the fastest setting, and run the SigLIP convolution layer on the CPU in parallel with the transformer blocks on the NPU to hide 200 ms of latency. A downsampler merges every 2 by 2 patch into one token, cutting 729 tokens per tile to 196, and the prefill runs in chunks of 128 tokens rather than all at once. Precision is mixed, with INT4 for the language model weights, INT8 for the vision encoder and projector weights and for the KV cache, and INT16 and FP16 activations, which the authors say keeps accuracy stable because activations are more sensitive to quantisation than weights. Against the 8B MiniCPM-V 2.5 running on the CPU of the same chip through llama.cpp, the authors report 2.53 s to load and encode a 768 by 1536 image and 2.7 s of prefill, versus 4.0 s and 13.9 s, at 24.4 tok/s against 4.9 tok/s. The paper describes no public release of weights or code, and the authors say future work will target a broader range of mobile devices. Source: https://arxiv.org/abs/2411.10640 Read the article: https://llmobile.kavents.com/ticker/bluelm-v/MediaTekNPUQuantisationResearchBenchmarkshttps://llmobile.kavents.com/ticker/phonelm/https://llmobile.kavents.com/ticker/phonelm/Thu, 07 Nov 2024 03:19:00 +0100Researchers at Beijing University of Posts and Telecommunications published PhoneLM on November 7, 2024, a small language model family in 0.5B and 1.5B sizes built in an unusual order. The team first searched transformer configurations for the fastest inference on a specific phone, then pre-trained the configuration that won, instead of choosing a shape for capability and optimising speed after the fact. PhoneLM-1.5B decodes at 58 tok/s on the CPU of a Xiaomi 14 with a Qualcomm Snapdragon 8 Gen 3. The search varied the layer count from 15 to 25, the ratio of intermediate to hidden size between 2 and 5 and the attention type, and the paper lists measured throughput for seven candidate 1.5B shapes on that phone. The winner has 19 layers and a hidden size of 2,560, a hidden size to layer ratio of 134.7 against the 54.6 to 88.6 the authors observe in recent small models. It uses ReLU rather than the SiLU activation most current small models adopt, which the authors argue is cheaper on NPUs built for integer arithmetic and leaves more zeros in the feed-forward output for sparse-computation tricks. The paper notes that only the 1.5B model went through the search. Across seven zero-shot tasks including HellaSwag, PIQA, BoolQ and both ARC sets, the benchmark table puts PhoneLM-1.5B at an average of 67.31%, ahead of SmolLM-1.7B at 66.49% and StableLM 2 1.6B at 66.34% and behind DCLM-1.4B at 68.33% and Gemma 2 2B at 71.31%. PhoneLM-0.5B averages 60.43% against 60.17% for SmolLM-360M and 56.46% for OpenELM-450M. The authors measure PhoneLM-1.5B decoding 1.2 times faster than StableLM 2 1.6B and 1.6 times faster than SmolLM 1.7B on the same handset. The two models saw 1.1T and 1.5T training tokens, all of them from open datasets such as DCLM-baseline, StarCoderData and Dolma. The group ran every device measurement through its own inference engine mllm, with the linear and embedding weights quantised to 4-bit and four threads pinned to the phone’s performance cores. On the NPU it goes through Qualcomm’s QNN framework, where the authors report prompt processing for PhoneLM-1.5B at 654 tok/s on the Xiaomi 14 against 602 tok/s for Qwen 2.5 1.5B. The 4-bit mllm build the group publishes is a 1.41 GB file. The paper states no peak memory figure. An Android demo app on top of mllm runs PhoneLM-1.5B locally for chat and for Android intent invocation, so a user who asks to be woken at 8 o’clock has an alarm set through the system’s own alarm intent. That variant, PhoneLM-1.5B-Call, was fine-tuned with LoRA on DroidCall, 10,000 synthetic function-calling examples covering alarms, timers, email drafts and searches, and the paper places it above GPT-4o-mini, which scores 71.0% on the same test set. Weights are on Hugging Face under Apache 2.0 and the data preparation, training, fine-tuning and evaluation code is on GitHub under GPL-2.0, alongside the list of open datasets used at each stage. Source: https://arxiv.org/abs/2411.05046 Read the article: https://llmobile.kavents.com/ticker/phonelm/ResearchOpen weightsAndroidNPUAgentshttps://llmobile.kavents.com/ticker/amd-llama-135m/https://llmobile.kavents.com/ticker/amd-llama-135m/Fri, 27 Sep 2024 19:00:00 +0200AMD released AMD-Llama-135M on September 27, 2024, its first small language model, at 135M parameters and trained from scratch on AMD’s own accelerators. AMD states that pretraining used 670B tokens drawn from SlimPajama and Project Gutenberg and took six days on four Instinct MI250 nodes holding four MI250 accelerators each. A second variant, AMD-Llama-135M-code, adds 20B tokens from the Python subset of StarCoder over another four days on the same hardware. That code variant exists to work as a draft model for speculative decoding, where a small model guesses several tokens in one pass and a larger model then checks them, so the larger model can accept several tokens per forward pass rather than producing one at a time. AMD paired it with Meta’s CodeLlama-7b and measured throughput on the HumanEval coding benchmark. AMD reports about 2.8x higher throughput on an Instinct MI250 accelerator, about 3.88x on the CPU of a Ryzen AI processor, and about 2.98x with the 7B model running 4-bit on the Ryzen AI NPU while the draft model ran on the CPU, each against the same setup without speculative decoding. AMD’s footnotes name a Ryzen 9 PRO 7940HS as the test machine for the Ryzen AI numbers. Diagram: AMD, showing its own measured speedups. AMD published the weights on Hugging Face and the training, dataset and inference code on GitHub, under the Apache 2.0 license. AMD states the training code builds on TinyLlama and uses PyTorch FSDP for distributed training across nodes. Source: https://www.amd.com/en/developer/resources/technical-articles/introducing-amd-first-slm-135m-model-fuels-ai-advancements.html Read the article: https://llmobile.kavents.com/ticker/amd-llama-135m/AMDLlamaNPUOpen weightsOpen sourcehttps://llmobile.kavents.com/ticker/fast-on-device-llm-inference-with-npus/https://llmobile.kavents.com/ticker/fast-on-device-llm-inference-with-npus/Mon, 08 Jul 2024 14:20:00 +0200Researchers at Peking University and the Beijing University of Posts and Telecommunications published llm.npu on July 8, 2024, an inference engine that runs the prefill stage of a language model on a phone’s NPU. They tested it on a Redmi K70 Pro with a Snapdragon 8 Gen 3 and a Redmi K60 Pro with a Snapdragon 8 Gen 2, using Qwen1.5-1.8B, Gemma-2B, Phi-2 2.7B, LLaMA2-7B and Mistral-7B. On 1024-token prompts on the K70 Pro the authors report prefill 7.3x to 38.4x faster than the CPU engines MNN and llama.cpp and 1.27x to 43.6x faster than the GPU engines TFLite and MLC-LLM, and for Qwen1.5-1.8B they measure 1,106 tok/s against 151 tok/s for MNN on the CPU and 57 tok/s for llama.cpp on the CPU. Energy was measured on the K60 Pro, the only device they could root, where prefill used 1.85x to 59.5x less energy than the same baselines. Chart: Xu et al., published under CC BY 4.0. The prompts come from the LongBench 2WikiMultihopQA dataset. Prefill is the stage where the model reads the whole prompt before it writes its first token, and the paper puts it at 88.3% to 98.8% of total latency on a mobile CPU for UI automation, chat summaries and context-aware generation, and at 54.2% to 91.7% on a mobile GPU. A mobile NPU is fast at integer matrix multiplication but expects fixed tensor shapes, while prompts arrive at any length, so every new length means rebuilding the graph. Quantised activations also contain outliers, single values far outside the usual range, and the accurate remedy is per-group quantisation, which mobile NPUs cannot execute directly and which the authors measure at up to 10.7x overhead when it is split into sub-tensor multiplications. llm.npu answers both by cutting a prompt into fixed 256-token chunks that run on pre-built graphs, sharing the operators that do not depend on prompt length so the memory cost drops by up to 4x, pulling the outlier values out to the CPU or GPU to run in parallel, and scheduling transformer blocks across the processors out of order. Accuracy stays within 1% of FP16 on LAMBADA, HellaSwag, WinoGrande, OpenBookQA and MMLU. Across three application workloads, UI automation, context-aware email replies and chat summaries, the authors measure end-to-end latency 1.4x to 32.8x lower than the baselines, with decoding still running unoptimised on the CPU. The system is built on the group’s own mllm inference framework and Qualcomm’s QNN SDK, in about 10,000 lines of C, C++ and assembly, and the code is public in the mllm repository under an MIT license. The paper appeared at ASPLOS 2025 in Rotterdam in spring 2025. PowerInfer-v2, the one comparison system that also uses the NPU for prefill, is not open source, so the authors took its figures from its paper and report a 3.28x to 5.6x speedup over it. Source: https://arxiv.org/abs/2407.05858 Read the article: https://llmobile.kavents.com/ticker/fast-on-device-llm-inference-with-npus/NPUQualcommAndroidQuantisationResearchhttps://llmobile.kavents.com/ticker/powerinfer-2/https://llmobile.kavents.com/ticker/powerinfer-2/Mon, 10 Jun 2024 16:01:21 +0200Researchers at the IPADS lab of Shanghai Jiao Tong University published PowerInfer-2 on June 10, 2024, an inference framework for running language models that exceed a phone’s memory. On a OnePlus 12, a Snapdragon 8 Gen 3 device with 24 GB of DRAM of which 19 GB can be occupied by an app, the authors measure TurboSparse-Mixtral-47B decoding at 11.68 tokens per second, and describe it as the first system to serve a 47B model on a smartphone. In the same configuration they report 3.12 times the speed of LLMFlash and 21.2 times the speed of llama.cpp. LLMFlash here is their own reimplementation of Apple’s LLM in a flash techniques inside llama.cpp, which they built because that work was never released as code. PowerInfer-2 splits each feedforward layer into neuron clusters, small groups of neurons that tend to activate together, and treats a cluster rather than a whole matrix as the unit of work. Frequently activated clusters go to the phone’s NPU for dense arithmetic, while rarely activated ones go to the CPU, which copes better with irregular sparse work. Weights that are not already in an in-memory cache are read from UFS flash on demand, and those reads are overlapped with computation so that a core can work on one cluster while the next is still arriving. For prompt processing the system loads whole layers with sequential reads, which the authors measure at 3 times the throughput of random reads, and prefetches the next layer while the current one computes. The authors vary how much memory the app may use from 7 GB to 19 GB with the 47B model. At 7 GB the cache holds only 1.8 percent of the feedforward weights, so nearly every neuron has to be fetched from flash for every token and decoding falls to 2.13 tokens per second, which is still 1.84 times LLMFlash. The 47B model activates only about 3B parameters per token thanks to its mixture-of-experts structure and added sparsity, which is why it decodes at roughly the pace of a 7B model in the same test. Streaming weights off storage to fit a large model into a small budget is the same bet Edge0 makes at the level of whole experts rather than individual neurons. Chart: Xue et al. LLMFlash is the authors' own reimplementation of Apple's method. On OpenCompass at 4-bit weights, the authors put PowerInfer-2 at an average of 78.38 across Arc-Challenge, Arc-Easy, MMLU and GSM8K for Qwen2 7B against 79.25 for llama.cpp, and at 68.35 against 70.12 for Bamboo-7B, with Qualcomm’s QNN engine at 56.93 and 63.26. Token latency is uneven, and for the 47B model with half the feedforward weights offloaded they report a mean of 99.76 ms per token and a 99th percentile of 140.56 ms, tracking a cache miss rate that averages 3.5 percent but reaches 18.9 percent at the 99th percentile. Prompt processing for Bamboo-7B runs at 404.6 tokens per second with half the weights offloaded, against over 700 tokens per second with everything in memory. They measure energy at 0.257 joules per token against 0.373 for QNN and 0.672 for llama.cpp. PowerInfer-2 uses privileged system calls such as memory locking, so the authors built and ran it on rooted Android phones, and they state that nothing in the design requires kernel changes and that it should port to iOS. It also needs models with predictable sparsity, which is why the group trained sparsified TurboSparse versions of Mistral and Mixtral, and on an unmodified SiLU-based Mistral-7B the gain over LLMFlash drops to 2.4 times against 4.6 times for the ReLU-based Bamboo-7B. The system is 12,000 lines of C++ added to the group’s PowerInfer engine for PCs, which is MIT-licensed, but the announcement there links the paper and the model weights rather than a release of the PowerInfer-2 code. The paper was last revised in December 2024. Source: https://arxiv.org/abs/2406.06282 Read the article: https://llmobile.kavents.com/ticker/powerinfer-2/MemoryNPUAndroidQualcommResearchhttps://llmobile.kavents.com/ticker/mobile-foundation-model-as-firmware/https://llmobile.kavents.com/ticker/mobile-foundation-model-as-firmware/Wed, 29 May 2024 15:00:00 +0200Researchers at Beijing University of Posts and Telecommunications proposed that a phone ship one shared multimodal model instead of letting every app bundle its own, in a paper published in the ACM MobiCom 2024 proceedings on May 29, 2024. The operating system and the hardware co-manage that model like firmware, unchangeable by apps or by the OS itself, exposed to applications as a system service, and each app reaches it through a small adapter fine-tuned offline for its own task. Their prototype, called M4, holds 9.2B parameters and needs 7.5 GB of peak memory, and the authors report it reaching accuracy comparable to purpose-built models on 85% of the 50 datasets in a benchmark they assembled from 38 mobile AI tasks across five input types. What the shared model replaces is one small model per app per task. The paper’s baselines are 50 task-specific models of 1M to 500M parameters each, one per dataset, against which M4’s adapters run from 1,000 to 10 million parameters, so each added task costs under 10 MB. Measured on an Nvidia Jetson Orin NX, 4-bit M4 needs 6.1 GB of storage to serve all 50 tasks against 15.2 GB for the 50 separate models, with the crossover at about 15 tasks, and 7.5 GB of peak memory against roughly five times that. The authors state that on a device with 12 GB of memory the 4-bit model plus all 50 adapters fits, where only 20 of the 50 task-specific models would. The prototype is slower than the models it replaces. On the Jetson Orin NX with 16 GB, the authors measured M4 averaging 18 times the inference latency of the task-specific models across the 50 tasks and 19 times the energy, 3.6 s against 0.2 s. On a Pixel 7 Pro CPU they measured an average of 6.8 s against 0.54 s, and their per-task breakdown puts image classification at 2.10 s and question answering at 6.34 s to the first token and 0.24 s per token after it. They state that M4 cannot currently run on a stock smartphone GPU or NPU at all, because those processors lack support for the operators it uses. The NPU numbers in the paper are a projection rather than a measurement. The authors estimate that M4 on an NPU would average 0.48 s and 1.3 J, under the 0.54 s and 2.9 J they measured for task-specific models on the Pixel 7 Pro CPU, but they derive that by applying the CPU-to-NPU ratio they observed for task-specific models, not by running M4 on an NPU. Their case for a simpler accelerator rests on a separate Pixel 7 Pro measurement, where they converted 110 downloaded models to TensorFlow Lite and only 8% ran entirely on the NPU, those gaining a median speedup above 20 times over the CPU. M4 itself uses 39 operator types against the 156 that the 50 task-specific models need between them. The authors name their own limits. They write that the accuracy results come from an A100 and the Jetson board rather than from phones, that M4 underperforms task-specific models on some tasks including translation, and that a prototype assembled from off-the-shelf pre-trained models is “still highly inefficient in terms of accuracy and model parameter size”. Its backbone is Meta’s LLaMA-7B at 8-bit, with encoders taken from ImageBind and Whisper, and they note that adapters trained against one backbone stop working when the backbone is upgraded, so the design still needs a stable interface between the two. Code and benchmark are published at github.com/UbiquitousLearning/MobileFM, and the paper carries ACM copyright rather than an open license. Source: https://dl.acm.org/doi/10.1145/3636534.3649361 Read the article: https://llmobile.kavents.com/ticker/mobile-foundation-model-as-firmware/ResearchBenchmarksNPUPixelLlamahttps://llmobile.kavents.com/ticker/qualcomm-ai-hub/https://llmobile.kavents.com/ticker/qualcomm-ai-hub/Mon, 26 Feb 2024 09:00:00 +0100Qualcomm launched Qualcomm AI Hub at Mobile World Congress in Barcelona on February 26, 2024, a developer service built around a model library of more than 75 pre-optimised AI and generative AI models for Snapdragon and Qualcomm platforms. Qualcomm states that spreading each model across the NPU, CPU and GPU of its AI Engine results in 4X faster inferencing times. The second half of the service is measurement. A developer uploads a trained model, the hub compiles it for a chosen chip and runtime, then runs it on a physical Snapdragon device in Qualcomm’s cloud and returns latency, memory use and a numerical check against the original, as the documentation describes. The same measurements are published for every catalogue entry, which is why each model page names the handset the figures came from. Diagram: Qualcomm. At launch the export recipes in Qualcomm’s repository covered two runtimes, TensorFlow Lite and Qualcomm AI Engine Direct, on Android 11 and newer, in FP16, INT8 and INT4. The supported chips ran from the Snapdragon 845 through the Snapdragon 888 to the Snapdragon 8 Gen 3, with Samsung Galaxy S21 to S24, Xiaomi 12 and 13 and Google Pixel 3 to 5 handsets named as targets. Everything is driven from a Python package called qai_hub. Two of the listed models generate text, Llama-v2-7B-Chat and Baichuan-7B, both quantised to 4-bit weights and 16-bit activations. Qualcomm’s model card puts Llama-v2-7B-Chat at 3.6 GB with a maximum context of 1024 tokens, and reports it measured on a Samsung Galaxy S23 Ultra running Android 13 on a Snapdragon 8 Gen 2, on the NPU, at 117.8 ms for each additional token and 2,578.5 ms to push a full 1024-token prompt through before the first token comes back. Peak memory for the token generator is 66 to 238 MB. Qualcomm publishes the models on the AI Hub site, on GitHub and on Hugging Face, and names no price for the service, asking only for a Qualcomm ID to sign in and an API token to submit jobs. The licensing is layered. Qualcomm’s export and optimisation code is BSD 3-Clause, each model keeps the licence of the original implementation, so Llama-v2-7B-Chat stays under Meta’s terms, and the compiled assets prepared for on-device deployment carry a separate Qualcomm AI Hub Proprietary License. Source: https://aihub.qualcomm.com/mobile/models Read the article: https://llmobile.kavents.com/ticker/qualcomm-ai-hub/QualcommDeveloper toolsAndroidNPULlamahttps://llmobile.kavents.com/ticker/snapdragon-8-gen-3/https://llmobile.kavents.com/ticker/snapdragon-8-gen-3/Tue, 24 Oct 2023 19:00:00 +0200Qualcomm launched the Snapdragon 8 Gen 3 at its Snapdragon Summit in Hawaii on October 24, 2023, calling it its first mobile platform designed with generative AI in mind. On the product page the company states support for generative models with up to 10 billion parameters on device and up to 20 tokens per second for large language models. Qualcomm puts the Hexagon NPU at 98 percent faster than the previous generation and claims 40 percent better performance per watt for sustained AI inference. It lists the Kryo CPU at 30 percent more performance and 20 percent better power efficiency, and the Adreno GPU at 25 percent faster rendering with 25 percent better efficiency. Among the workloads Qualcomm names for local execution are Stable Diffusion, virtual assistants and chatbots. Devices were announced by ASUS, Honor, iQOO, MEIZU, Nubia, OnePlus, OPPO, realme, Redmi, RedMagic, Sony, vivo, Xiaomi and ZTE. Image: Qualcomm. Source: https://www.qualcomm.com/news/releases/2023/10/qualcomm-launches-premium-snapdragon-8-gen-3-to-bring-generative Read the article: https://llmobile.kavents.com/ticker/snapdragon-8-gen-3/QualcommChipsNPUAndroidhttps://llmobile.kavents.com/ticker/qualcomm-stable-diffusion-android/https://llmobile.kavents.com/ticker/qualcomm-stable-diffusion-android/Thu, 23 Feb 2023 17:00:00 +0100Qualcomm AI Research has demonstrated Stable Diffusion v1.5, a model with more than 1 billion parameters, running entirely on an Android phone powered by a Snapdragon 8 Gen 2. The company reports image generation in under 15 seconds for 20 inference steps at 512x512 pixels, a workload that until then ran in the cloud. Images: Qualcomm. Sample outputs from the demo, generated on the phone. To fit the model on the phone, Qualcomm quantised it from FP32 to INT8 using its AI Model Efficiency Toolkit and a post-training method called AdaRound, which adapts how individual weights are rounded. According to the company, the model needed no retraining after the conversion. Inference runs through the Qualcomm AI Engine direct framework on the Hexagon processor, with the model prepared in Qualcomm AI Studio. The company sums up the result in its post. This is the fastest inference on a smartphone and comparable to cloud latency. Source: https://www.qualcomm.com/news/onq/2023/02/worlds-first-on-device-demonstration-of-stable-diffusion-on-android Read the article: https://llmobile.kavents.com/ticker/qualcomm-stable-diffusion-android/QualcommImage generationQuantisationAndroidNPU