https://llmobile.kavents.com/tags/pytorch/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/mobilellm-flash/https://llmobile.kavents.com/ticker/mobilellm-flash/Mon, 16 Mar 2026 23:10:00 +0100Meta AI published MobileLLM-Flash on March 16, 2026, three models of 350M, 650M and 1.4B parameters whose shapes were picked by measuring how fast each candidate actually ran on a phone. Meta reports the family reaching up to 1.8 times faster prefill and 1.6 times faster decode than Liquid AI’s LFM2 models on mobile CPUs, at comparable or higher accuracy, with support for up to 8k tokens of context. Meta’s first MobileLLM paper argued in 2024 that deep and thin networks win below a billion parameters, and the Flash paper states that such deep-and-thin structures often fail to improve on-device latency. Meta exported 100 architectures to a phone and found parameter count a poor predictor of measured prefill and decode speed, at a Kendall rank correlation of 0.40, with FLOPs little better at 0.46 and 0.55. The 1.4B model therefore runs 16 layers at a width of 2048, against the 30 layers at 1280 of MobileLLM-Pro-1B. Meta ran the latency tests on a Samsung Galaxy S25, a phone with a Snapdragon 8 Elite chipset, an octa-core CPU and 12 GB of memory, exporting each model through ExecuTorch to the XNNPACK CPU backend at 4-bit weights, 8-bit dynamic activations and a quantised KV cache, on 4 CPU threads and averaged over three runs after a warmup. At a 2k context the 650M model reaches its first token in 3.34 s against 6.01 s for LFM2 700M, and decodes at 85.35 tok/s against 53.57 tok/s. Meta repeated the 1k-context test on an iPhone 17, where the 650M model takes 2.81 s to the first token against 3.40 s for LFM2 700M. Most of the speed comes from letting some layers skip attention, so a token passes through those blocks without being compared to every earlier token. The 350M model keeps full attention in 7 of its 12 layers and the 650M model in 8 of 13, while the 1.4B model attends in all 16. Meta reports that skipping consistently beat sliding window attention, which instead limits each token to a fixed span of recent ones, and that more than three skipped layers in a row costs accuracy at the same latency, dropping TriviaQA exact match from 33.2% to 8.8%. Meta built the three models by pruning a shallow 1.8B variant of MobileLLM-Pro and inheriting its weights, which the paper says took about 35% of the tokens that pretraining MobileLLM-Pro required. Across nine reasoning and knowledge tasks Meta reports averages of 45.46, 48.57 and 55.06 for the three sizes, against 44.92, 48.52 and 50.48 for LFM2 at 350M, 700M and 1.2B, 51.79 for Gemma 3 1B and 49.01 for Llama 3.2 1B. After instruction tuning, the 1.4B model scores 47.89 on MMLU and 46.34 on HumanEval, against 49.30 and 37.80 for Llama 3.2 1B. The paper announces no release of the weights. Chart: Meta. Bubble area shows parameter count, latency measured on a Samsung Galaxy S25. Source: https://arxiv.org/abs/2603.15954 Read the article: https://llmobile.kavents.com/ticker/mobilellm-flash/MetaResearchBenchmarksAndroidPyTorchhttps://llmobile.kavents.com/ticker/executorch-1-0/https://llmobile.kavents.com/ticker/executorch-1-0/Wed, 22 Oct 2025 18:00:00 +0200The PyTorch team released ExecuTorch 1.0 on October 22, 2025, the generally available version of its runtime for PyTorch models on phones, laptops and embedded devices. It takes a model straight from PyTorch without converting it to ONNX or TFLite and without rewriting it in another language, which the team says removes a common source of numerical mismatches and lost debug information. The beta arrived in October 2024. Diagram: PyTorch documentation. Five backends moved from beta to production status. XNNPACK with Arm Kleidi covers CPU, Apple Core ML covers Apple silicon, the Qualcomm AI Engine delegate covers the Hexagon NPU, and Arm Ethos-U and Vulkan cover NPU and GPU. New backends arrived for Arm VGF, the NXP eIQ Neutron NPU, Samsung Exynos NPU and GPU, and Intel OpenVINO. Meta says features built on ExecuTorch already run on Instagram, WhatsApp, Messenger and Facebook. Anuj Kumar, Director at Reality Labs, describes what it drives in Meta’s wearables, including the Ray-Ban Display glasses with EMG band. Advanced AI features like speech recognition, motion sensing, and computer vision run on these devices by leveraging cutting-edge hardware. Lysandre Debut, Chief Open-Source Officer at Hugging Face, says more than 80 percent of the most downloaded edge-friendly language models on Hugging Face run on ExecuTorch out of the box. The team validated coverage across object detection, depth, OCR, speech recognition and segmentation as well as text and multimodal models such as Voxtral for audio and Gemma 3 for image and text input. The main branch also allows embedding ExecuTorch into native C++ desktop applications. Source: https://pytorch.org/blog/introducing-executorch-1-0/ Read the article: https://llmobile.kavents.com/ticker/executorch-1-0/MetaPyTorchDeveloper toolsNPUOpen sourcehttps://llmobile.kavents.com/ticker/small-model-training-bottlenecks/https://llmobile.kavents.com/ticker/small-model-training-bottlenecks/Fri, 25 Oct 2024 12:30:00 +0200Seven Apple researchers measured what it costs to train language models up to 2B parameters on rented Nvidia GPUs and found the most expensive card was the worst buy. The H100-80GB returned the fewest tokens per dollar of the three cards tested at every one of their four model sizes, 100M, 500M, 1B and 2B, while costing 2.53 times as much per hour as the A100-40GB. The price scale comes from normalising published hourly rates at Lambda Labs and Google Cloud, with the A100-40GB set at 1.00 and the A100-80GB landing at 1.46. The same comparison shows cost efficiency falling as GPUs are added, with a single card the best or near-best point at all four model sizes. For the 1B model, tokens per dollar drop from about 17,000 on one GPU to about 10,300 on 64. FlashAttention mattered more at small scale than at large. Reading the authors’ first figure, the 100M model gained roughly three times the tokens per dollar over vanilla attention, while at 2B the gain was under two times. The authors put this down to the cost of attention being quadratic in context length, so it dominates once the hidden dimension shrinks and the small models become bound by data movement between CPU and GPU and between GPUs. Vanilla attention also ran out of memory at a global batch size of 1024 on the 1B and 2B models, where FlashAttention did not. Two habits carried over from large-model training did not pay off. Plain Distributed Data Parallel, which keeps a full copy of the model on every GPU, beat both sharded FSDP variants for the smaller models because it moves less data, and FSDP only pulled ahead at 2B, where it kept training at a global batch size that made DDP run out of memory. Filling the cards was not cost-optimal either, since the batch size figure peaks around a global batch size of 128 for the 100M model, 32 for 500M, 16 for 1B and 8 for 2B, in every case well short of what the memory allowed. These are training runs on datacentre GPUs, not anything running on a phone, and the authors frame the work as guidance for labs with small budgets rather than for deployment. They name their own limits, running each setup at least three times for only 10 training steps, reporting normalised price ratios rather than absolute dollars, using no gradient accumulation, and leaving GPU utilisation and memory bandwidth measurements to future work. They also assume that settings such as the learning rate can be retuned to converge at whichever hardware configuration turns out cheapest. The 8-page paper is free to read on arXiv under the site’s non-exclusive distribution licence. Source: https://arxiv.org/abs/2410.19456 Read the article: https://llmobile.kavents.com/ticker/small-model-training-bottlenecks/AppleNvidiaResearchBenchmarksPyTorchhttps://llmobile.kavents.com/ticker/torchchat/https://llmobile.kavents.com/ticker/torchchat/Tue, 30 Jul 2024 18:00:00 +0200PyTorch released torchchat on July 30, 2024, a small codebase showing how to run Llama 3 and other language models on laptops, desktops and phones. For the phone path the team reports more than 8 tok/s for Llama 3 8B Instruct on a Samsung Galaxy S23 and on iPhone, using 4-bit GPTQ quantisation through ExecuTorch. Diagram: PyTorch. The repository names iOS 17 or later with at least 8 GB of RAM as its requirement, which it narrows to iPhone 15 Pro and newer or an iPad with Apple silicon, and on Android any device covered by the XNNPACK CPU backend, with 12 GB of RAM and 20 GB of storage recommended. Its model table marks Llama 3.2 1B and 3B, Llama 3.1 8B, Llama 2 7B, Mistral 7B, IBM Granite 3.1 and DeepSeek R1 Distill 8B as mobile friendly. The 11B vision version of Llama 3.2 and the 13B and 70B Llama 2 checkpoints carry no such mark. Getting a model onto a phone is one export step that turns a downloaded checkpoint into a .pte file for ExecuTorch, the PyTorch edge runtime that reached version 1.0 in October 2025. The mobile preset shipped in the repository quantises the embedding table to 4-bit in groups of 32 values and the linear layers to 4-bit weights whose activations are quantised to 8-bit at runtime, in groups of 256. The quantisation guide offers 4-bit and 8-bit embeddings at group sizes from 32 to 256 across all execution modes, lists the dynamic-activation scheme as the supported one for ExecuTorch, and notes that smaller groups preserve accuracy while larger ones run faster. Both demo apps are sample harnesses rather than shipped products. On iOS the repository points at the ExecuTorch LLaMA example project in Xcode, which needs a provisioning profile carrying Apple’s increased memory limit entitlement, after which the model and tokenizer files are copied into the app’s folder on the device. On Android the repository carries its own app skeleton for Android Studio, built against a prebuilt ExecuTorch library, with the model and tokenizer pushed to the phone over adb. The other two paths in torchchat do not touch phones. One runs the model in Python through a chat and generate command line, a browser interface or a local server, the other compiles it ahead of time with PyTorch’s AOTInductor into a binary that a C++ runner loads on a desktop or server. The code is under a BSD 3-Clause license, and PyTorch halted active development on May 19, 2025, saying vLLM had become the dominant option for server-side inference. Source: https://github.com/pytorch/torchchat Read the article: https://llmobile.kavents.com/ticker/torchchat/PyTorchLlamaAndroidiOSQuantisationhttps://llmobile.kavents.com/ticker/executorch-alpha/https://llmobile.kavents.com/ticker/executorch-alpha/Tue, 30 Apr 2024 18:00:00 +0200PyTorch released ExecuTorch alpha on April 30, 2024, the first version of its edge runtime aimed at large language models rather than small vision and speech networks. PyTorch says the work enables Llama 2 7B to run on iPhone 15 Pro, iPhone 15 Pro Max and Samsung Galaxy S22, S23 and S24 phones, with early support for Llama 3 8B. The example project shipped with the release reports 8.15 to 11.6 tok/s for the quantised 7B model, measured by PyTorch over adb on a Galaxy S22, a Galaxy S24 and a OnePlus 12. The 0.1 preview from October 2023 had no such path. Fitting a 7B model on a phone came down to quantisation. Every linear layer carries 4-bit weights in groups of 128 or 256 values, with activations quantised to 8-bit at runtime from their observed range. PyTorch measured WikiText perplexity at 9.16 for the full-precision model against 10.2 and 10.7 for the two group sizes, and notes that groups smaller than 128 were not enabled because the resulting files were still too large, since the embedding table and the weight scales remained in 32-bit floats. Support for 16-bit floats was under way. Arm, Apple and Qualcomm Technologies worked on the release, and PyTorch describes the mobile accelerator paths through Core ML, MPS, TOSA and the Qualcomm AI Stack as ongoing work rather than finished. The release notes add Snapdragon 8 Gen 3 support and 4-bit and 16-bit quantisation on the Qualcomm side, over 100 operators on Apple’s MPS backend, and a Vulkan delegate for mobile GPUs. PyTorch also credits Google for joint work on XNNPACK, the CPU library that carried the language model numbers, and says MediaTek was enabling the Llama models on its own chips. PyTorch was explicit about what alpha did not yet do. The release notes mark XNNPACK as the only finished performance path for generative models and everything else as in progress, and the announcement calls on-device LLM support early while asking the community for help building GPU and NPU delegates and expanding the quantisation schemes. Meta was already shipping the runtime elsewhere, using it for hand tracking on Quest 3, for several models on the Ray-Ban Meta smart glasses and in a beginning rollout inside Instagram. The Python package became installable from PyPI with this release. Source: https://pytorch.org/blog/executorch-alpha/ Read the article: https://llmobile.kavents.com/ticker/executorch-alpha/PyTorchLlamaQuantisationAppleQualcomm