https://llmobile.kavents.com/tags/developer-tools/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/gemini-nano-4-mlkit-prompt-api/https://llmobile.kavents.com/ticker/gemini-nano-4-mlkit-prompt-api/Wed, 22 Jul 2026 18:00:00 +0200Google’s Android developer blog states that Samsung’s new foldable devices come with Gemini Nano 4, which it calls its latest on-device model. The post credits Nano 4 with support for more than 140 languages and better multimodal understanding. Apps reach the model through ML Kit’s Prompt API, which sends natural language requests on-device to Gemini Nano. It takes text, or a combination of image and text, and returns text or structured output. Google names structured output and thinking mode as the features to use for on-device intelligence. The ML Kit release notes dated 14 July 2026 record the structured output API, system instructions and thinking mode arriving in the Prompt API, along with multi-image support and an output token limit raised to 4,096 tokens. A note dated 21 July records a fix for Gemini Nano v4 compatibility in the Prompt API on non-Pixel devices. The Prompt API moved from alpha to beta in January 2026 and carries no service level agreement or deprecation policy. The post also points developers to app functions, which share an app’s capabilities with the Gemini Intelligence system. Its remaining sections cover adaptive layouts, fold-aware design, CameraX and Wear OS widgets. Source: https://android-developers.googleblog.com/2026/07/optimize-galaxy-screen-sizes.html Read the article: https://llmobile.kavents.com/ticker/gemini-nano-4-mlkit-prompt-api/GoogleAndroidSamsungGemini NanoDeveloper toolshttps://llmobile.kavents.com/ticker/apple-foundation-models-sdk/https://llmobile.kavents.com/ticker/apple-foundation-models-sdk/Wed, 25 Feb 2026 20:47:00 +0100Apple published python-apple-fm-sdk, Python bindings for its Foundation Models framework, with a first beta release on February 25, 2026 according to the repository’s release history. The package reaches the on-device model at the core of Apple Intelligence from Python on macOS, and installs with pip install apple-fm-sdk. The Python API follows the Swift one. SystemLanguageModel reports whether the model is available and, if not, why, while LanguageModelSession holds a conversation and its async respond call returns text or streams it. A @fm.generable decorator marks a Python class for the model to fill in, and fm.guide adds per-field constraints such as a numeric range, as the README shows with a Cat class whose age is bounded to 0 through 20. Tool calling and session transcripts are exposed as well. The bindings do not reimplement inference. The repository carries a Swift package named foundation-models-c that wraps the framework behind a C interface and builds it as a dynamic library, which the Python layer then calls. Apple writes in the documentation that the bindings run the Swift framework underneath, so evaluations reflect real on-device performance and behaviour, and its evaluation guide notes that inference calls are processed one at a time rather than in parallel at the macOS hardware level. Apple lists macOS 26.0 or later, Xcode 26.0 or later with its agreement accepted in the Xcode app, Python 3.10 or later, and Apple Intelligence switched on. PyPI carries only source distributions for the package, so the Swift component is compiled locally during installation, which is what the Xcode requirement covers. Apple Intelligence on the Mac runs on Apple silicon machines only. The code is under the Apache 2.0 license and the package metadata on PyPI classifies it as alpha. Apple pushed 0.1.0 to PyPI on March 8, 2026, added the Attachment API for sending images alongside text in 0.2.0 on June 8, 2026 for WWDC 2026, and exposed the model’s context size and the token count of a given input in 0.2.1 on June 29, 2026. The README states the project is not yet taking contributions. Source: https://github.com/apple/python-apple-fm-sdk Read the article: https://llmobile.kavents.com/ticker/apple-foundation-models-sdk/AppleDeveloper toolsApple SiliconOpen sourcehttps://llmobile.kavents.com/ticker/executorch-1-0/https://llmobile.kavents.com/ticker/executorch-1-0/Wed, 22 Oct 2025 18:00:00 +0200The PyTorch team released ExecuTorch 1.0 on October 22, 2025, the generally available version of its runtime for PyTorch models on phones, laptops and embedded devices. It takes a model straight from PyTorch without converting it to ONNX or TFLite and without rewriting it in another language, which the team says removes a common source of numerical mismatches and lost debug information. The beta arrived in October 2024. Diagram: PyTorch documentation. Five backends moved from beta to production status. XNNPACK with Arm Kleidi covers CPU, Apple Core ML covers Apple silicon, the Qualcomm AI Engine delegate covers the Hexagon NPU, and Arm Ethos-U and Vulkan cover NPU and GPU. New backends arrived for Arm VGF, the NXP eIQ Neutron NPU, Samsung Exynos NPU and GPU, and Intel OpenVINO. Meta says features built on ExecuTorch already run on Instagram, WhatsApp, Messenger and Facebook. Anuj Kumar, Director at Reality Labs, describes what it drives in Meta’s wearables, including the Ray-Ban Display glasses with EMG band. Advanced AI features like speech recognition, motion sensing, and computer vision run on these devices by leveraging cutting-edge hardware. Lysandre Debut, Chief Open-Source Officer at Hugging Face, says more than 80 percent of the most downloaded edge-friendly language models on Hugging Face run on ExecuTorch out of the box. The team validated coverage across object detection, depth, OCR, speech recognition and segmentation as well as text and multimodal models such as Voxtral for audio and Gemma 3 for image and text input. The main branch also allows embedding ExecuTorch into native C++ desktop applications. Source: https://pytorch.org/blog/introducing-executorch-1-0/ Read the article: https://llmobile.kavents.com/ticker/executorch-1-0/MetaPyTorchDeveloper toolsNPUOpen sourcehttps://llmobile.kavents.com/ticker/lm-meter/https://llmobile.kavents.com/ticker/lm-meter/Tue, 07 Oct 2025 19:05:00 +0200Researchers at Georgia State University and Toyota InfoTech Labs published lm-Meter, a latency profiler that runs inside the inference engine on the phone and splits each generation into embedding, prefill, decode, softmax and sampling. Measuring the Pythia models on a Google Pixel 8 Pro, they report that scaling from 70M to 1.4B parameters raises prefill latency from 0.012 s to 1.9 s per input token, a 158x slowdown, while decode latency per output token grows from 0.015 s to 0.15 s, a 10x slowdown. The authors write that this inverts the server picture, where decode is usually the limiting phase for single-request inference. Below the phase level the profiler times individual GPU kernels through OpenCL event timestamps, which give queue, submit, start and end times without access to the closed-source driver. Running a 4-bit quantised Gemma-2-2B-it on a Pixel 8 Pro, the authors report that fused matrix-multiplication kernels dominate a decode step and that the GPU sits idle for more than 21% of it, the second-largest contributor to the step, which they attribute to host-side data preparation and I/O stalls. The paged attention kernel that scans the growing key-value cache is the only one whose cost rises with position in the sequence, climbing from roughly 0.2 ms to about 0.8 ms per token over 250 decode steps, and idle time drops from about 21% to 12% when the model generates 256 tokens instead of 16. Whether those measurements mean anything depends on what the profiler itself costs. lm-Meter sits in the MLC LLM runtime and TVM in about 3,500 lines of code and needs no host machine attached, and under the Powersave CPU governor, the most constrained setting they tested, the authors measure a throughput loss of 2.58% in prefill and 0.99% in decode. They put the same figures for MELTing Point, the on-device profiler they compare against, at 22% for prefill and more than 93% for decode. Checked against traces from Android GPU Inspector, they report end-to-end phase accuracy of at least 99.99% and mean kernel-level accuracy of 96.82% on the Pixel 8 Pro and 96.61% on a Pixel 7. The code is on GitHub under the MIT license, with the MLC LLM path released for Android GPUs through OpenCL and support for llama.cpp, vLLM, iOS Metal and Nvidia Jetson listed as unfinished. The work was accepted to the ACM/IEEE Symposium on Edge Computing 2025 and funded by Toyota Motor North America. The measurements come from three phones, the Pixel 8 Pro, Pixel 7 and Pixel 6, and the authors state that other edge platforms such as Jetson boards and Intel NPUs may show different bottlenecks. Source: https://arxiv.org/abs/2510.06126 Read the article: https://llmobile.kavents.com/ticker/lm-meter/ResearchBenchmarksDeveloper toolsMLC LLMPixelhttps://llmobile.kavents.com/ticker/apple-foundation-models-framework/https://llmobile.kavents.com/ticker/apple-foundation-models-framework/Mon, 09 Jun 2025 21:00:00 +0200Apple opened its on-device foundation model to third-party apps at WWDC25 on June 9, 2025. The Foundation Models framework gives any app direct access to the roughly 3-billion-parameter model, which runs offline and costs developers nothing per call. Developers annotate Swift data structures with the @Generable macro, and the framework applies constrained decoding, which Apple calls guided generation, so the model returns those structures rather than free text. Tool calling lets the model invoke functions the app supplies. Developers who need more control can train rank 32 adapters with a Python toolkit. Apple compressed the model to 2 bits per weight using quantisation-aware training. It splits the transformer into two blocks at a 5:3 depth ratio and shares the key-value cache between them, which Apple says cuts KV cache memory use by 37.5 percent. Images are handled by a 300M-parameter ViTDet-L vision encoder. The model is designed to support 15 languages. The companion server model on Private Cloud Compute uses what Apple calls a Parallel-Track Mixture-of-Experts design, where independent transformer tracks cut how often the tracks have to synchronise with each other. Diagram: Apple. The server model, not the on-device one. Source: https://machinelearning.apple.com/research/apple-foundation-models-2025-updates Read the article: https://llmobile.kavents.com/ticker/apple-foundation-models-framework/AppleiOSDeveloper toolsQuantisationAgentshttps://llmobile.kavents.com/ticker/flower-intelligence/https://llmobile.kavents.com/ticker/flower-intelligence/Wed, 12 Mar 2025 10:00:00 +0100Flower Labs released Flower Intelligence on March 12, 2025 as a preview, an open source inference library that runs a language model on the user’s own device and only moves the work to a remote service if the app and the user allow it. The company shipped SDKs for Swift and TypeScript, and the TypeScript build defaults to Llama 3.2 1B in 16-bit precision. The library picks a local engine from the environment it finds itself in. The TypeScript package depends on MLC’s WebLLM, which executes models in the browser through WebGPU, and on Hugging Face transformers.js, which it uses when the code runs under Node.js rather than in a page. The Swift package is built on MLX Swift and Hugging Face swift-transformers, and declares iOS 16 and macOS 14 as its minimum platforms, so local inference on Apple hardware goes through MLX. Flower Labs’ model reference covers Llama 3.2 at 1B and 3B, Llama 3.1 8B, SmolLM2 at 135M, 360M and 1.7B, and DeepSeek R1 Distill Llama 8B, most of them as 4-bit builds. Which of those run on device depends on the backend. Node.js is limited to the 1B and SmolLM2 entries, while the 8B models run only in the browser and on the MLX Swift path, and Llama 3.2 3B in 16-bit is MLX Swift only. Remote handoff is off by default. An application has to set the remoteHandoff property and supply an API key before the library will contact Flower Confidential Remote Compute, and even with both in place a chat call tries a local engine first and retries remotely only when no local engine matches the model or the local run fails. Individual requests can be pinned with forceLocal or forceRemote. Flower Labs describes the remote service as a private extension of the device that costs the developer no extra work, and named Mozilla Thunderbird as an early adopter building its Thunderbird Assist feature on the library. Ryan Sipes, Managing Director for Mozilla Thunderbird, tied that choice to the mail client’s user base in the announcement. Our 20 million users expect data privacy from every feature we build. Both SDKs are published under the Apache 2.0 license, the TypeScript one as @flwr/flwr on npm and the Swift one as a package in the Flower repository. Flower Labs called the release inference-only and listed fine-tuning, RAG and pre-training as later steps, with the remote compute service available on application for early access. Source: https://flower.ai/blog/2025-03-12-flower-intelligence Read the article: https://llmobile.kavents.com/ticker/flower-intelligence/Open sourceDeveloper toolsLlamaWebGPUApple Siliconhttps://llmobile.kavents.com/ticker/llama-stack/https://llmobile.kavents.com/ticker/llama-stack/Wed, 25 Sep 2024 19:30:00 +0200Meta announced Llama Stack at Connect on September 25, 2024, a set of REST APIs plus client SDKs for Python, Node, Swift and Kotlin that cover the same Llama features wherever the model runs. The repository defines APIs for Inference, Safety, Memory, Agentic System, Evaluation, Post Training, Synthetic Data Generation and Reward Scoring, and lists PyTorch ExecuTorch as the on-device provider, covering inference and agents on iOS. The models announced at the same event are covered separately. Llama Stack separates providers from distributions. A provider is the implementation behind an API, which can be local code or a pointer to a remote service, and a distribution assembles providers behind one endpoint. Meta’s README states that providers can be mixed, that a hobbyist can serve a small model locally and use a cloud provider for a large one, and that the higher level APIs the app works with do not change, including across the server and mobile-device boundary. Diagram: Meta. From the Llama Stack repository documentation. On iOS the local provider is a Swift class called LocalInference, which the documentation describes as powered by ExecuTorch. It implements the same Inference interface the Swift client uses for remote servers, so an app that already streams chat completions or agent turns keeps its calling code and swaps the service it was given. Developers export the model to an ExecuTorch .pte file, bundle it with tokenizer.model in the app, and link the XNNPACK, Core ML and MPS backends plus the quantized and optimised kernels, with Meta’s own example loading a SpinQuant build of Llama 3.2 1B. The Swift package is added through Xcode’s package manager and generates its types from the Llama Stack OpenAPI spec, and Meta listed its first release on October 10, 2024. The same iOS documentation stated that on-device inference for Android was coming soon, and pointed developers to ExecuTorch’s own Android demo app in the meantime. Llama Stack, the Swift client and the Kotlin client are published under the MIT license. Update, December 6, 2024. Meta released version 0.0.54 of the Kotlin client, which adds on-device inference on Android. The README names ExecuTorch as the local inference distributor, has developers run a script that fetches a prebuilt executorch.aar with the XNNPACK delegate into the app’s libs directory, and shows the only difference between the two modes as the client builder, LlamaStackClientLocalClient with a model and tokenizer path against LlamaStackClientOkHttpClient with a base URL. The chatCompletion call that follows is identical. Source: https://github.com/meta-llama/llama-stack Read the article: https://llmobile.kavents.com/ticker/llama-stack/MetaLlamaiOSAndroidDeveloper toolshttps://llmobile.kavents.com/ticker/mediapipe-llm-inference-api/https://llmobile.kavents.com/ticker/mediapipe-llm-inference-api/Thu, 07 Mar 2024 17:00:00 +0100Google released the MediaPipe LLM Inference API on March 7, 2024, an experimental way to run language models fully on device from web, Android and iOS apps. Four open models are supported at launch, Gemma 2B, Phi 2, Falcon 1B and Stable LM 3B. The API quantises weights to int8, with Gemma 2B using mixed 4-bit and 8-bit weights. Google measured throughput on unnamed high-end devices with a 1024-token input prompt and a maximum of 1280 tokens. In its charts, Gemma 2B at int4 prefills at roughly 680 tokens per second on WebGPU and on an Android GPU, and decodes at about 57 tokens per second on WebGPU against 31 on an Android GPU and 27 on iOS. Chart: Google. Chart: Google. Gemma 2B at int4 was the only model that ran on iOS. Google marks the Android version as intended for experimental and research use only, and points production apps to the Gemini API or to Gemini Nano through Android AICore instead. On iOS, Gemma 2B at int4 was the only model the team could run, which Google attributes to the memory available on the platform. Source: https://developers.googleblog.com/en/large-language-models-on-device-with-mediapipe-and-tensorflow-lite/ Read the article: https://llmobile.kavents.com/ticker/mediapipe-llm-inference-api/GoogleMediaPipeDeveloper toolsAndroidiOShttps://llmobile.kavents.com/ticker/qualcomm-ai-hub/https://llmobile.kavents.com/ticker/qualcomm-ai-hub/Mon, 26 Feb 2024 09:00:00 +0100Qualcomm launched Qualcomm AI Hub at Mobile World Congress in Barcelona on February 26, 2024, a developer service built around a model library of more than 75 pre-optimised AI and generative AI models for Snapdragon and Qualcomm platforms. Qualcomm states that spreading each model across the NPU, CPU and GPU of its AI Engine results in 4X faster inferencing times. The second half of the service is measurement. A developer uploads a trained model, the hub compiles it for a chosen chip and runtime, then runs it on a physical Snapdragon device in Qualcomm’s cloud and returns latency, memory use and a numerical check against the original, as the documentation describes. The same measurements are published for every catalogue entry, which is why each model page names the handset the figures came from. Diagram: Qualcomm. At launch the export recipes in Qualcomm’s repository covered two runtimes, TensorFlow Lite and Qualcomm AI Engine Direct, on Android 11 and newer, in FP16, INT8 and INT4. The supported chips ran from the Snapdragon 845 through the Snapdragon 888 to the Snapdragon 8 Gen 3, with Samsung Galaxy S21 to S24, Xiaomi 12 and 13 and Google Pixel 3 to 5 handsets named as targets. Everything is driven from a Python package called qai_hub. Two of the listed models generate text, Llama-v2-7B-Chat and Baichuan-7B, both quantised to 4-bit weights and 16-bit activations. Qualcomm’s model card puts Llama-v2-7B-Chat at 3.6 GB with a maximum context of 1024 tokens, and reports it measured on a Samsung Galaxy S23 Ultra running Android 13 on a Snapdragon 8 Gen 2, on the NPU, at 117.8 ms for each additional token and 2,578.5 ms to push a full 1024-token prompt through before the first token comes back. Peak memory for the token generator is 66 to 238 MB. Qualcomm publishes the models on the AI Hub site, on GitHub and on Hugging Face, and names no price for the service, asking only for a Qualcomm ID to sign in and an API token to submit jobs. The licensing is layered. Qualcomm’s export and optimisation code is BSD 3-Clause, each model keeps the licence of the original implementation, so Llama-v2-7B-Chat stays under Meta’s terms, and the compiled assets prepared for on-device deployment carry a separate Qualcomm AI Hub Proprietary License. Source: https://aihub.qualcomm.com/mobile/models Read the article: https://llmobile.kavents.com/ticker/qualcomm-ai-hub/QualcommDeveloper toolsAndroidNPULlamahttps://llmobile.kavents.com/ticker/gemini-nano-pixel-8-pro/https://llmobile.kavents.com/ticker/gemini-nano-pixel-8-pro/Wed, 06 Dec 2023 18:00:00 +0100Google brought Gemini Nano to the Pixel 8 Pro in its December 2023 feature drop, where it powers Summarize in Recorder and Smart Reply in Gboard. Google calls the Pixel 8 Pro the first smartphone engineered for Gemini Nano and runs the model on the Tensor G3. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: Google. Gemini Nano summarising a recording in the Recorder app. Open the Android Developers post According to Google, running the model locally helps prevent sensitive data from leaving the phone and lets the features work without a network connection. Summarize in Recorder launched in English. Smart Reply in Gboard launched globally on the United States English keyboard layout, starting with WhatsApp, Line and KakaoTalk. On the same day Google introduced AICore, a system service in Android 14 that handles model management, runtimes and safety features for Gemini Nano. It supports Low Rank Adaptation, so developers can build small adapters trained on their own data, and it targets the Google Tensor TPU as well as NPUs from Qualcomm, Samsung and MediaTek. Google describes the service as isolated from the network by design and opened access through an early access programme. Diagram: Google. Source: https://blog.google/products/pixel/pixel-feature-drop-december-2023/ Read the article: https://llmobile.kavents.com/ticker/gemini-nano-pixel-8-pro/GoogleGemini NanoPixelAndroidDeveloper toolshttps://llmobile.kavents.com/ticker/apple-mlx/https://llmobile.kavents.com/ticker/apple-mlx/Tue, 05 Dec 2023 01:04:00 +0100Apple machine learning research shipped the first public release of MLX on December 5, 2023, an array framework for machine learning on Apple silicon. The repository carries a v0.0.2 tag dated that day and the same version went up on PyPI minutes later, after an initial commit on November 28. Arrays in MLX live in shared memory, so an operation can run on the CPU or on the GPU without the data being copied from one place to the other. That behaviour comes from the hardware. Apple silicon uses a unified memory architecture in which the CPU and the GPU have direct access to the same memory pool, and the MLX documentation states that an array therefore has no device of its own. Code names the device when it runs an operation instead of moving an array to a device first, and when two operations on different devices depend on each other, the documentation says the MLX scheduler inserts the dependency between the streams automatically. Computation is lazy. The documentation states that operations only record a compute graph and that nothing is computed until an eval call asks for a result, which is what lets MLX apply function transformations such as automatic differentiation and automatic vectorisation to the graph. It gives memory as the second reason, since a model whose weights are created as float32 and then replaced with float16 before any evaluation peaks at half the memory eager computation would need. The MLX Swift bindings followed a week later, with an initial commit on December 12, 2023, and the package manifest lists macOS 14, iOS 17, tvOS 17 and visionOS 1 as supported platforms. The example apps build for iOS as well as macOS and include a chat client for language and vision-language models, a text generation demo that downloads weights from Hugging Face, and Stable Diffusion image generation. The LLMEval example uses the Increased Memory Limit entitlement on iOS, which its README attributes to the size of language model weights, and caps the MLX buffer cache at 20 MB. MLX LM is the Python package that runs and fine-tunes language models on top of MLX, first published to PyPI on January 12, 2024. It installs the mlx_lm.generate and mlx_lm.chat command line tools, whose default model is a 4-bit quantised Llama 3.2 3B Instruct from the mlx-community organisation on Hugging Face, and it can quantise and upload converted models back to the Hub. MLX is published under the MIT license, and the install documentation lists Apple silicon, macOS 14.0 or newer and Python 3.10 or newer for the macOS package, alongside CUDA and CPU-only builds for Linux added later. Source: https://github.com/ml-explore/mlx Read the article: https://llmobile.kavents.com/ticker/apple-mlx/AppleApple SiliconDeveloper toolsOpen sourceiOShttps://llmobile.kavents.com/ticker/llm-swift/https://llmobile.kavents.com/ticker/llm-swift/Mon, 04 Dec 2023 00:11:00 +0100The South Korean developer eastriverlee published the first release of LLM.swift late on December 3, 2023, a Swift package that wraps llama.cpp so an iOS or Mac app can load a GGUF model file and generate text on the device in a few lines. The launch example creates a model object from a file bundled with the app and awaits a reply that streams back token by token into a SwiftUI view. The package declared macOS 12, iOS 14, watchOS 4, tvOS 14 and visionOS 1 as its minimum versions, and required Swift 5.9. Screenshot: LLM.swift. The test app shipped in the first release README. Sitting on llama.cpp means the package reads GGUF files and nothing else, and the project states that any model llama.cpp runs should run here too, which is a different route from Hugging Face’s swift-transformers and its Core ML pipeline. Chat formatting was left to the app author at first, a closure that assembled the prompt plus a stop string that ends generation, and the release seven weeks later replaced that with ready-made templates for ChatML, Alpaca and Llama, with Mistral and Gemma added since. A developer adds the package through the Swift Package Manager by pointing at the repository, and the README recommends models of 3B parameters or fewer on mobile devices, reporting a Mistral 7B model running on an iPad Air 5th generation at 5-bit quantisation and on an iPhone 12 mini at 2-bit. The project has since put out 41 further releases, the most recent in July 2026, and version 3 made the chat template embedded in the GGUF file the default, added function calling and schema-constrained structured output, and switched to a prebuilt llama.cpp framework fetched from an upstream release by URL and checksum. The minimum versions have risen to iOS 16, macOS 13, watchOS 9, tvOS 16 and visionOS 1. Almost all of the roughly 150 commits are eastriverlee’s own, and the package is published under the MIT license. Source: https://github.com/eastriverlee/LLM.swift Read the article: https://llmobile.kavents.com/ticker/llm-swift/Developer toolsiOSllama.cppOpen sourceApple Siliconhttps://llmobile.kavents.com/ticker/tinychatengine/https://llmobile.kavents.com/ticker/tinychatengine/Thu, 24 Aug 2023 22:45:00 +0200Song Han’s group at MIT pushed the first commit of TinyChatEngine on August 24, 2023, according to the repository’s commit history. The library runs quantised language models in plain C and C++ with no dependency on bulky libraries, and the README names x86 CPUs from Intel and AMD, ARM devices including Apple M1 and M2 Macs and the Raspberry Pi, and Nvidia GPUs through CUDA. The lab’s own slides put LLaMA2-7B at 71 ms per token on an M1 Pro and 210 ms per token on an Intel i7-9750H laptop CPU. The compression behind it comes from two papers by the same lab. AWQ, for activation-aware weight quantisation, was submitted on June 1, 2023 and works from the finding that protecting only 1 percent of the weights, picked by looking at the activation distribution rather than the weights themselves, keeps most of the accuracy at 4 bits. SmoothQuant, submitted on November 18, 2022, handles 8-bit weights and 8-bit activations by shifting the hard part of the job from activations to weights through a mathematically equivalent transformation, and its authors report up to 1.56x speedup and 2x memory reduction. Diagram: MIT HAN Lab. The backend table lists W4A32, W4A8 and W8A8 on both x86 and ARM CPUs, W4A16 on Nvidia GPUs, and FP32 on CPU. TinyChatEngine reorders the 4-bit weights offline during model conversion rather than at runtime, in a layout tailored to each instruction set, with QM_ARM built for 128-bit SIMD and QM_x86 for 256-bit SIMD. The slides state that expanding the 4-bit weights to int8 and using int8 SIMD multiply-accumulate instead of FP32 gives 1.3x on the Intel i7-9750H and 3x on the M1 Pro. For the device comparison the lab measured LLaMA2-7B against PyTorch and llama.cpp. On the Intel i7-9750H it reports 210 ms per token for TinyChatEngine, 233 for llama.cpp and 1375 for PyTorch, which it labels a 6.5x gap. On an Nvidia Jetson Orin the figures are 108, 133 and 141 ms per token, and on the M1 Pro TinyChatEngine lands at 71 ms per token against 66 for llama.cpp, with PyTorch running out of memory. The lab publishes no Raspberry Pi timing. Chart: MIT HAN Lab's own figures, from the project slides. The model zoo ships LLaMA-7B, LLaMA2-7B-chat and LLaMA2-13B-chat plus OPT at 125M, 1.3B and 6.7B, in FP32, AWQ int4 and SmoothQuant int8 variants, with a download script that takes a per-device quantisation flag. macOS users install boost and llvm through Homebrew, and Xcode for the Metal compiler that enables Apple GPU support. The repository carries an MIT license, added the day after the first commit. For the Raspberry Pi the project recommends a board with 8 GB of RAM and says its testing ran on a Raspberry Pi 4 Model B Rev 1.4 with aarch64. Source: https://github.com/mit-han-lab/TinyChatEngine Read the article: https://llmobile.kavents.com/ticker/tinychatengine/QuantisationApple SiliconResearchOpen sourceDeveloper toolshttps://llmobile.kavents.com/ticker/hf-swift-transformers/https://llmobile.kavents.com/ticker/hf-swift-transformers/Tue, 08 Aug 2023 16:15:00 +0200Hugging Face published swift-transformers on August 8, 2023, the date of the repository’s first tag, a Swift package that gives iOS and Mac developers a transformers-like API for running language models on the device through Core ML. Pedro Cuenca introduced it as an in-development library built around four modules, Tokenizers, Hub, Generation and Models. The Tokenizers module ports the normalisation and pre-processing rules that the Python tokenizers library keeps as configuration files next to a model on the Hub, so an app does not hardcode those decisions per model and can swap one checkpoint for another. It covered BPE at release, the family used by the GPT models, Falcon and Llama, with Unigram and WordPiece listed as later work. The Hub module downloads those configuration files and the model metadata, and Generation implements greedy decoding and top-k sampling. The Models module wraps a converted Core ML package behind a LanguageModel type, and Core ML then decides whether to run it on the CPU, the GPU or the Neural Engine. Cuenca writes that flexible input shapes run on the CPU only, which is why Hugging Face’s exporters conversion tool switched to fixed sequence lengths, and that the Core ML path cached no attention keys and values at that point, both of which cap how long a context an app can work with. Hugging Face states it tested the package with Llama 2, Falcon, StarCoder and the GPT family including GPT2, distilgpt, GPT-NeoX and GPT-J, and that encoder-decoder models such as T5 and Flan were not supported. The release came alongside swift-chat, a Mac app that loads a Core ML model and doubles as a model tester, plus converted weights for Llama 2 7B and Falcon 7B on the Hub. The package is licensed under Apache 2.0. Screenshot: Hugging Face. The demo app exposes the generation parameters and reports throughput. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: Hugging Face. swift-chat generating a haiku from Llama 2 7B on an M1 MacBook Pro. Open the Hugging Face post Source: https://github.com/huggingface/swift-transformers Read the article: https://llmobile.kavents.com/ticker/hf-swift-transformers/Developer toolsiOSApple SiliconOpen sourceLlamahttps://llmobile.kavents.com/ticker/llama-cpp-first-release/https://llmobile.kavents.com/ticker/llama-cpp-first-release/Fri, 10 Mar 2023 20:50:00 +0100Georgi Gerganov pushed the first commits of llama.cpp on March 10, 2023, according to the repository’s commit history. The project is an inference engine for Meta’s LLaMA models written in plain C and C++, and the README states its goal as language model and vision-language model inference with minimal setup and state-of-the-art performance on a wide range of hardware, locally and in the cloud. The core implementation carries no external dependencies. It supports integer quantisation at 1.5, 2, 3, 4, 5, 6 and 8 bits, which is what brings models within reach of consumer memory budgets. Apple silicon is a first-class target through ARM NEON, the Accelerate framework and Metal. The project has since added backends for x86 with AVX, AVX2, AVX512 and AMX, RISC-V, NVIDIA CUDA, AMD HIP, Moore Threads MUSA, Vulkan, SYCL, OpenCL and WebGPU. It is published under the MIT license. Source: https://github.com/ggml-org/llama.cpp Read the article: https://llmobile.kavents.com/ticker/llama-cpp-first-release/llama.cppOpen sourceQuantisationApple SiliconDeveloper toolshttps://llmobile.kavents.com/ticker/tinygrad/https://llmobile.kavents.com/ticker/tinygrad/Sun, 18 Oct 2020 07:57:00 +0200George Hotz pushed the first commit of tinygrad on October 18, 2020, with the message “start tinygrad”, according to the repository’s commit history. The README describes an end-to-end deep learning stack with a tensor library, an autograd, a compiler that fuses and lowers kernels, and a JIT, positioned between PyTorch and karpathy/micrograd. It also states that an accelerator port needs a total of about 25 low level ops. tiny corp, the company Hotz founded, maintains it. The README lists backends for OpenCL, CPU, Metal, CUDA, AMD, NVIDIA, QCOM and WebGPU. The runtime documentation names the requirements for each one, Adreno 6xx series GPUs for the QCOM backend and M1 or newer Macs for Metal, with Metal 3.0 needed for bfloat support. The QCOM backend, which drives the Qualcomm GPU through the Adreno kernel driver interface instead of OpenCL, entered the repository on September 2, 2024, and the repository also carries a backend for Qualcomm’s Hexagon DSP in ops_dsp.py. tiny corp states on its site that tinygrad is used in openpilot to run the driving model on the Snapdragon 845 GPU, where it replaces Qualcomm’s SNPE. The company says its framework is faster than SNPE, loads ONNX files, supports training and allows attention, which SNPE does not because it only allows fixed weights. The repository’s test workflow compiles an openpilot model with an image pitch alignment set to match the Adreno 630, the GPU in the Snapdragon 845. The same workflow enforces the project’s size, failing the build when the repository goes over 26,500 lines as counted by sz.py. The repository ships a language model tool in tinygrad/llm that loads GGUF files, serves an OpenAI compatible API and benchmarks tokens per second, with a built-in model list that starts at quantised Llama 3.2 1B and Qwen3 0.6B. tinygrad is published under the MIT license, and the project recommends installing from source rather than from PyPI. Diagram: tinygrad. Source: https://github.com/tinygrad/tinygrad Read the article: https://llmobile.kavents.com/ticker/tinygrad/Open sourceDeveloper toolsQualcommApple SiliconWebGPUhttps://llmobile.kavents.com/ticker/onnx-runtime/https://llmobile.kavents.com/ticker/onnx-runtime/Tue, 29 Sep 2020 19:00:00 +0200Microsoft released ONNX Runtime 1.5 on September 29, 2020 with a preview of ORT for Mobile Platforms, described in the release notes as minimising build size for mobile and embedded devices. The same release added a reduced operator kernel build, which compiles the runtime with only the operators the target models actually use, and the ORT model format that those reduced builds require. The format documentation records version 1 of the ORT format as introduced in 1.5. Diagram: ONNX Runtime documentation. The pre-built mobile packages, onnxruntime-android for Java, C and C++ and onnxruntime-c and onnxruntime-objc for iOS, carry the full operator set and load .onnx files directly, while a custom build takes models converted to the .ort format. Microsoft’s mobile guide puts numbers on the difference for version 1.18.0, where a custom build supporting the operators of a ResNet50 model produces an Android AAR of 7,532,309 bytes against 24,415,212 bytes for the pre-built package, and an arm64-v8a libonnxruntime.so of 3,962,832 bytes against 16,276,832 bytes. Accelerators are called execution providers. The NNAPI provider requires Android 8.1 or higher and is recommended on Android 9 or higher, the Core ML provider requires iOS 13 or macOS 10.15 and higher, and the XNNPACK provider covers floating-point CPU work on both platforms. Microsoft’s guidance is to start on the CPU provider for a quantised model and on XNNPACK otherwise, then try NNAPI or Core ML if that misses the performance target. The QNN provider reaches Qualcomm Snapdragon chips on Android and Windows through the Qualcomm AI Engine Direct SDK, with pre-built packages for Windows only and an HTP backend that takes quantised models only. Quantisation in ONNX Runtime is 8-bit linear quantisation, offered as dynamic and static post-training quantisation in two representations, QOperator with dedicated quantised operators and QDQ with QuantizeLinear and DequantizeLinear pairs inserted around the originals. The mobile guide states that moving 32-bit weights to 8 bits cuts model size by roughly a factor of 4. The tooling also does block-wise weight-only quantisation to int4 and uint4 for MatMul and Gather nodes, with RTN, HQQ and GPTQ algorithms, and models with GatherBlockQuantized nodes need ONNX Runtime 1.20 or later to run. Generative models run through the separate onnxruntime-genai package, which implements the generation loop, KV cache management, search and sampling around the runtime. Its support matrix lists Android as supported and iOS as on the roadmap, with CPU, QNN, WebGPU, CUDA, DirectML and OpenVINO among the acceleration options. Microsoft reported in April 2024 that RTN int4 quantisation shrinks Phi-3 Mini enough to run on a Samsung Galaxy S21 at what it called a moderate speed, and recommends the int4_accuracy_level=4 variant on mobile. Both projects are published under the MIT license. Source: https://github.com/microsoft/onnxruntime/releases/tag/v1.5.1 Read the article: https://llmobile.kavents.com/ticker/onnx-runtime/MicrosoftAndroidiOSQuantisationDeveloper tools