https://llmobile.kavents.com/categories/apps/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/rikkahub-agent-local-llm/https://llmobile.kavents.com/ticker/rikkahub-agent-local-llm/Sat, 08 Aug 2026 00:00:00 +0200Adam Conway at XDA has tested RikkaHub Agent, a fork by ExTV of the open-source Android LLM client RikkaHub, which adds an agent layer with more than 80 device tools. The phone runs the agent loop and executes the tools; the model itself runs elsewhere on the local network over an OpenAI-compatible endpoint. The test device was an Oppo Find N5 with a Snapdragon 8 Elite and 16 GB of RAM. Inference ran on a Radeon RX 7900 XTX with Qwen 3.6 27B at Q4_K_M, roughly 16 GB, at 40 to 50 tokens per second. A second configuration used Qwen 3.6 35B on a Lenovo ThinkStation PGX with Nvidia’s GB10 Grace Blackwell. Screenshot: XDA. The tool set covers tapping, swiping, scrolling and typing on screen, reading notifications, sending SMS, file management, browser control, SSH, scheduled jobs, a JavaScript runtime and Telegram. All tools are off by default and enabled per assistant, actions with side effects require approval, and a deterministic blocklist rejects destructive commands. The default context is 7,000 input tokens, rising to 32,000 with every tool enabled. Asked whether whisper.cpp was installed, the agent determined that neither the command-line tool nor a model was present, requested permission, then installed the dependencies, cloned the repository and compiled it inside Termux. The build took seven minutes, and the agent chose the 75 MB small model on speed and accuracy grounds. It then transcribed a Telegram voice message, installing ffmpeg, converting the OGG/Opus file to 16 kHz mono WAV and returning a transcript in about two seconds. Running the model on the phone itself was slower. With Gemma 4 E2B, a 2.6 GB build accelerated by LiteRT-LM, Conway reports chat worked but agentic tasks were painful, and tool calls were emitted as plain text until an update fixed the behaviour. Frontier intelligence is never what I’d want from a phone agent, which is why a local LLM works perfectly. Source: https://www.xda-developers.com/turned-phone-ai-agent-powered-local-llm-installed-software/ Read the article: https://llmobile.kavents.com/ticker/rikkahub-agent-local-llm/AndroidAgentsQwenOpen sourceSpeechhttps://llmobile.kavents.com/ticker/gemini-nano-4-mlkit-prompt-api/https://llmobile.kavents.com/ticker/gemini-nano-4-mlkit-prompt-api/Wed, 22 Jul 2026 18:00:00 +0200Google’s Android developer blog states that Samsung’s new foldable devices come with Gemini Nano 4, which it calls its latest on-device model. The post credits Nano 4 with support for more than 140 languages and better multimodal understanding. Apps reach the model through ML Kit’s Prompt API, which sends natural language requests on-device to Gemini Nano. It takes text, or a combination of image and text, and returns text or structured output. Google names structured output and thinking mode as the features to use for on-device intelligence. The ML Kit release notes dated 14 July 2026 record the structured output API, system instructions and thinking mode arriving in the Prompt API, along with multi-image support and an output token limit raised to 4,096 tokens. A note dated 21 July records a fix for Gemini Nano v4 compatibility in the Prompt API on non-Pixel devices. The Prompt API moved from alpha to beta in January 2026 and carries no service level agreement or deprecation policy. The post also points developers to app functions, which share an app’s capabilities with the Gemini Intelligence system. Its remaining sections cover adaptive layouts, fold-aware design, CameraX and Wear OS widgets. Source: https://android-developers.googleblog.com/2026/07/optimize-galaxy-screen-sizes.html Read the article: https://llmobile.kavents.com/ticker/gemini-nano-4-mlkit-prompt-api/GoogleAndroidSamsungGemini NanoDeveloper toolshttps://llmobile.kavents.com/ticker/airgap/https://llmobile.kavents.com/ticker/airgap/Wed, 29 Apr 2026 08:06:11 +0200Xavier Puspus published Airgap on April 29, 2026, a React Native starter kit for branded customer support apps that have to keep answering when the network is slow, missing or deliberately blocked. Its configuration file pins a Gemma 4 E2B instruction-tuned model in Q3_K_S GGUF form, about 2.4 GB, which the app downloads on first launch and runs through llama.rn, the React Native binding for llama.cpp. The README states that Airgap searches bundled local documents first and uses a model only to phrase what that retrieval returned, and it lists Apple Foundation Models and Google’s ML Kit GenAI Prompt API as two further on-device answer providers an operator can put ahead of the downloaded file. The repository ships seven example configurations, for airline, banking, electric utility, healthcare, insurance, telecom and water utility support, each with its own branding, prompts and JSON knowledge files, and the downloaded-model path targets iOS 15.1 and newer and Android API 24 and newer. Airgap is published under the MIT license. It is an early single-author project, with 54 commits and no tagged release, and almost all of the code arrived in one squashed initial commit followed by a burst of work in August 2026. The README lists physical-device evaluation of the Apple and Android system providers as an open release gate and labels the published demo recordings as Android emulator and iOS simulator footage rather than evidence of either system model running on a phone. Source: https://github.com/xmpuspus/airgap Read the article: https://llmobile.kavents.com/ticker/airgap/Open sourceAndroidiOSGemmallama.cpphttps://llmobile.kavents.com/ticker/apple-foundation-models-sdk/https://llmobile.kavents.com/ticker/apple-foundation-models-sdk/Wed, 25 Feb 2026 20:47:00 +0100Apple published python-apple-fm-sdk, Python bindings for its Foundation Models framework, with a first beta release on February 25, 2026 according to the repository’s release history. The package reaches the on-device model at the core of Apple Intelligence from Python on macOS, and installs with pip install apple-fm-sdk. The Python API follows the Swift one. SystemLanguageModel reports whether the model is available and, if not, why, while LanguageModelSession holds a conversation and its async respond call returns text or streams it. A @fm.generable decorator marks a Python class for the model to fill in, and fm.guide adds per-field constraints such as a numeric range, as the README shows with a Cat class whose age is bounded to 0 through 20. Tool calling and session transcripts are exposed as well. The bindings do not reimplement inference. The repository carries a Swift package named foundation-models-c that wraps the framework behind a C interface and builds it as a dynamic library, which the Python layer then calls. Apple writes in the documentation that the bindings run the Swift framework underneath, so evaluations reflect real on-device performance and behaviour, and its evaluation guide notes that inference calls are processed one at a time rather than in parallel at the macOS hardware level. Apple lists macOS 26.0 or later, Xcode 26.0 or later with its agreement accepted in the Xcode app, Python 3.10 or later, and Apple Intelligence switched on. PyPI carries only source distributions for the package, so the Swift component is compiled locally during installation, which is what the Xcode requirement covers. Apple Intelligence on the Mac runs on Apple silicon machines only. The code is under the Apache 2.0 license and the package metadata on PyPI classifies it as alpha. Apple pushed 0.1.0 to PyPI on March 8, 2026, added the Attachment API for sending images alongside text in 0.2.0 on June 8, 2026 for WWDC 2026, and exposed the model’s context size and the token count of a given input in 0.2.1 on June 29, 2026. The README states the project is not yet taking contributions. Source: https://github.com/apple/python-apple-fm-sdk Read the article: https://llmobile.kavents.com/ticker/apple-foundation-models-sdk/AppleDeveloper toolsApple SiliconOpen sourcehttps://llmobile.kavents.com/ticker/off-grid-offline-ai-app/https://llmobile.kavents.com/ticker/off-grid-offline-ai-app/Sat, 14 Feb 2026 23:39:00 +0100Ali Chherawalla has posted Off Grid to Show HN, an Android and iOS app that runs text generation, image generation, vision and speech recognition entirely on the device. The submission reached 124 points and 66 comments. The repository is licensed under MIT. The app bundles llama.cpp for text, Stable Diffusion for images and whisper.cpp for speech to text. It accepts any GGUF file and lists Qwen 3, Llama 3.2, Gemma 3 and Phi-4 among supported models; vision runs on SmolVLM, Qwen3-VL or Gemma 3n. It also connects to OpenAI-compatible servers on a local network, and builds a project knowledge base from PDFs chunked and embedded on-device with a bundled MiniLM model into SQLite. The README reports 15 to 30 tokens per second on flagship CPUs and 20 to 40 tokens per second through Adreno GPUs via OpenCL on a Snapdragon 8 Gen 2 or newer. Image generation is given as 5 to 10 seconds on the Hexagon NPU and vision inference at about 7 seconds. The NPU path is marked experimental: it accelerates only Q4_0 and Q8_0 quantisation, and K-quants fall back to the CPU. Comments disputed whether models of that size are usable. One reader wrote: Ive tried a lot of self hosted models at a lot of sizes; those small models are not good enough, and do not have a context long enough to be useful for most everyday operations. Chherawalla replied that “you don’t need claude level performance for a lot of day to day tasks” and that Qwen3 8B, or 4B quantised, is “actually quite good”. Other threads compared the app with PocketPal AI, SmolChat and Google’s AI Edge Gallery, asked for an F-Droid build, and reported that image generation required network access on GrapheneOS. A reader on a Samsung device reported the interface did not move for the on-screen keyboard; the developer shipped a fix the same day. Source: https://news.ycombinator.com/item?id=47019133 Read the article: https://llmobile.kavents.com/ticker/off-grid-offline-ai-app/AndroidiOSOpen sourcellama.cppImage generationhttps://llmobile.kavents.com/ticker/executorch-1-0/https://llmobile.kavents.com/ticker/executorch-1-0/Wed, 22 Oct 2025 18:00:00 +0200The PyTorch team released ExecuTorch 1.0 on October 22, 2025, the generally available version of its runtime for PyTorch models on phones, laptops and embedded devices. It takes a model straight from PyTorch without converting it to ONNX or TFLite and without rewriting it in another language, which the team says removes a common source of numerical mismatches and lost debug information. The beta arrived in October 2024. Diagram: PyTorch documentation. Five backends moved from beta to production status. XNNPACK with Arm Kleidi covers CPU, Apple Core ML covers Apple silicon, the Qualcomm AI Engine delegate covers the Hexagon NPU, and Arm Ethos-U and Vulkan cover NPU and GPU. New backends arrived for Arm VGF, the NXP eIQ Neutron NPU, Samsung Exynos NPU and GPU, and Intel OpenVINO. Meta says features built on ExecuTorch already run on Instagram, WhatsApp, Messenger and Facebook. Anuj Kumar, Director at Reality Labs, describes what it drives in Meta’s wearables, including the Ray-Ban Display glasses with EMG band. Advanced AI features like speech recognition, motion sensing, and computer vision run on these devices by leveraging cutting-edge hardware. Lysandre Debut, Chief Open-Source Officer at Hugging Face, says more than 80 percent of the most downloaded edge-friendly language models on Hugging Face run on ExecuTorch out of the box. The team validated coverage across object detection, depth, OCR, speech recognition and segmentation as well as text and multimodal models such as Voxtral for audio and Gemma 3 for image and text input. The main branch also allows embedding ExecuTorch into native C++ desktop applications. Source: https://pytorch.org/blog/introducing-executorch-1-0/ Read the article: https://llmobile.kavents.com/ticker/executorch-1-0/MetaPyTorchDeveloper toolsNPUOpen sourcehttps://llmobile.kavents.com/ticker/apple-foundation-models-framework/https://llmobile.kavents.com/ticker/apple-foundation-models-framework/Mon, 09 Jun 2025 21:00:00 +0200Apple opened its on-device foundation model to third-party apps at WWDC25 on June 9, 2025. The Foundation Models framework gives any app direct access to the roughly 3-billion-parameter model, which runs offline and costs developers nothing per call. Developers annotate Swift data structures with the @Generable macro, and the framework applies constrained decoding, which Apple calls guided generation, so the model returns those structures rather than free text. Tool calling lets the model invoke functions the app supplies. Developers who need more control can train rank 32 adapters with a Python toolkit. Apple compressed the model to 2 bits per weight using quantisation-aware training. It splits the transformer into two blocks at a 5:3 depth ratio and shares the key-value cache between them, which Apple says cuts KV cache memory use by 37.5 percent. Images are handled by a 300M-parameter ViTDet-L vision encoder. The model is designed to support 15 languages. The companion server model on Private Cloud Compute uses what Apple calls a Parallel-Track Mixture-of-Experts design, where independent transformer tracks cut how often the tracks have to synchronise with each other. Diagram: Apple. The server model, not the on-device one. Source: https://machinelearning.apple.com/research/apple-foundation-models-2025-updates Read the article: https://llmobile.kavents.com/ticker/apple-foundation-models-framework/AppleiOSDeveloper toolsQuantisationAgentshttps://llmobile.kavents.com/ticker/googleai-edge-gallery/https://llmobile.kavents.com/ticker/googleai-edge-gallery/Sun, 18 May 2025 00:21:00 +0200Google’s AI Edge team has published Google AI Edge Gallery, an app that runs generative models entirely on the device. The home screen offers three tasks, AI Chat for multi-turn conversation, Ask Image for questions about a photo from the camera or the gallery, and Prompt Lab for single-turn work, which covers free-form prompts, rewriting text in a chosen tone, summarising and generating code. Once a model file is on the phone, the app needs no connection and Google states that all processing happens on the device. Screenshot: Google. The third screen shows the per-model settings, including the GPU and CPU toggle. Models come from the LiteRT Community organisation on Hugging Face, which the app’s welcome text links to directly. The first release listed three downloads, Gemma 3 1B IT at 4-bit in 554.7 MB, Hammer 2.1 1.5B at 8-bit and Qwen2.5 1.5B Instruct at 8-bit, the latter two at roughly 1.6 GB each. Gated repositories ask for a Hugging Face access token, which the settings dialog stores and can clear again. Anything outside that list has to arrive as a LiteRT .task bundle picked from device storage, and the app tells the user that no other file type is supported. Each model then carries a settings dialog with an accelerator switch between GPU and CPU, next to max tokens, top-k, top-p and temperature. The shipped list sets a different default per model, GPU first for Gemma 3 1B and CPU only for Qwen2.5 1.5B Instruct. After every reply the app prints four of its own measurements, time to first token in seconds, prefill speed and decode speed in tokens per second, and total latency in seconds. It derives them from the session itself, dividing the prompt’s token count by the time the first token took and counting decoded tokens against the clock that starts with that first token, so the numbers describe whatever hardware the app happens to be running on. Google calls this an “experimental Alpha release” and ships it under the Apache 2.0 licence. There is no store listing at this point, so installing means taking the APK from the project’s releases page on a device running Android 8.0 or newer, with a project wiki covering the corporate-device case. The README lists Android as available now and iOS as coming soon. Underneath, the app builds on LiteRT and Google’s MediaPipe LLM inference API for Android. Source: https://github.com/google-ai-edge/gallery Read the article: https://llmobile.kavents.com/ticker/googleai-edge-gallery/GoogleAndroidLiteRTHugging FaceOpen sourcehttps://llmobile.kavents.com/ticker/locally-ai/https://llmobile.kavents.com/ticker/locally-ai/Sun, 13 Apr 2025 09:00:00 +0200Adrien Grondin released Locally AI on April 13, 2025, an iPhone and iPad app that downloads open-weight language models and runs them entirely on the device. The App Store listing gives that release date and lists the app as free with no in-app purchases. Once a model is on disk the app needs no account and no internet connection, and Grondin states that no data is collected and nothing is sent to a cloud service. The site at launch named Meta Llama 3.2 and Llama 3.1, Google Gemma 2, Qwen 2 and DeepSeek R1 as the models on offer, and described them as language and vision models, so a picture can go into the prompt alongside text. A user picks a model, downloads it once and can edit the system prompt the model runs under. Grondin builds the app on MLX, Apple’s machine learning framework for Apple silicon, and attributes the performance to unified memory, where the CPU and the GPU reach the same memory pool so arrays need not be copied between them. The app fetches open-weight model files itself rather than calling any model that ships with the operating system. Neither the site nor the App Store entry names a minimum chip. The site says only that Locally AI covers recent iPhone and iPad models and is optimised for Apple silicon, and the listing asks for iOS 18.1 or iPadOS 18.1 or later. No generation speed, memory requirement or model file size is given anywhere in the developer’s own material, and the launch site was a bought template whose team, testimonial and FAQ sections still carried placeholder Latin text. Update, April 8, 2026. Grondin added support for Apple’s on-device Foundation Models framework in September 2025, so the app can open a conversation with the model built into iOS 26 without downloading anything, which Simon Willison recorded on September 21, 2025. A Mac version followed and requires macOS 26 or later. LM Studio announced on April 8, 2026 that Locally AI had joined the company and that Grondin had moved with it, and the App Store entry now reads Locally AI by LM Studio with Element Labs Inc as the seller. Source: https://locallyai.app/ Read the article: https://llmobile.kavents.com/ticker/locally-ai/AppleApple SiliconMLXiOSiPhonehttps://llmobile.kavents.com/ticker/huggingsnap/https://llmobile.kavents.com/ticker/huggingsnap/Wed, 19 Mar 2025 16:16:00 +0100Hugging Face published HuggingSnap, an iPhone app that describes what the camera is pointed at, with the model running on the phone rather than in the cloud. The app’s runtime configuration loads SmolVLM2 at 500M parameters in Apple’s MLX format. Hugging Face had trailed the app in its SmolVLM2 announcement on February 20, 2025, writing that the 500M model lets users analyse and understand video content directly on their device with no cloud required. The README describes the app as a way to point the camera and have text translated or summarised, identify plants and animals and learn more about surrounding objects. Inference runs through MLX Swift, Apple’s machine learning framework for Apple silicon, with a modified fork of Apple’s MLX example code supplying the vision-language support. The app pulls the weights from the Hugging Face Hub the first time it runs, then holds Metal memory to 82 percent of what iOS reports as available to the process. Input comes from the live camera, from stills and clips captured in the app, and from photos and videos picked out of the library. A tap on the shutter takes a picture and a press and hold records video, after which the app offers a describe action for images and a summarise action for video, with questions typed or dictated. The app’s own interface states that photos and videos are processed entirely on the device and that no data is sent to the cloud. The README states that installation needs an iPhone running iOS 18, and the Xcode project sets iOS 18.0 as the deployment target and iPhone as the only device family, with no Mac Catalyst or visionOS build. TechCrunch reported the app as a free App Store download on March 19, 2025, quoting the listing as saying it works offline and processes all data on the phone. The SmolVLM2 weights are Apache 2.0, while the app repository itself carries no licence file. Source: https://github.com/huggingface/HuggingSnap Read the article: https://llmobile.kavents.com/ticker/huggingsnap/Hugging FaceiOSiPhoneApple SiliconOpen weightshttps://llmobile.kavents.com/ticker/alibaba-mnn/https://llmobile.kavents.com/ticker/alibaba-mnn/Thu, 23 Jan 2025 13:30:00 +0100Alibaba published the first public build of its MNN-LLM Android app on January 23, 2025, a client that downloads open models and runs them entirely on the phone, covering text-to-text, image-to-text, audio-to-text and text-to-image generation through diffusion models. It sits on MNN, the inference and training framework Alibaba has kept open source since 2019 and uses in more than 30 of its own apps, among them Taobao, Youku and DingTalk. Alibaba reports the runtime prefilling 8.6 times faster than llama.cpp on an Android CPU, with decoding 2.3 times faster. Models reach the device through llmexport, which converts a PyTorch checkpoint to ONNX and then to MNN’s own format, splitting the weights into a separate file and quantising them to 4-bit or 8-bit by default, with block size configurable and 5-bit and 6-bit available through the MNNConvert tool. The same README lists Qwen, Gemma, Llama including TinyLlama and MobileLLM, Baichuan, Yi, DeepSeek, InternLM, Phi, ReaderLM and SmolLM as supported families. LoRA adapters can be merged into the base model at export or kept as a separate file that the runtime loads and swaps while running. The MNN-LLM paper, presented at the ACM Multimedia Asia 2024 workshops, describes a DRAM-Flash split that leaves the embedding table, roughly 15% of the weights, in bfloat16 on flash storage and reads only the single token needed for each decode step. The authors put the saving for Qwen2 7B at about 2.18 GB of DRAM. Layers and the LM head stay in memory at int4 or int8 with asymmetric quantisation, computed as W4A8 or W8A8 on CPU and W4A16 or W8A16 on GPU, while the KV cache quantises keys to int4 or int8 and values to fp8 before spilling to flash with prefetching. The team measured the speed comparisons on a Xiaomi 14 against llama.cpp, MLC-LLM and fastllm, using Qwen2 1.5B, Qwen2 7B and Llama 3 8B. Android builds run on the CPU with ARMv8.2 fp16 instructions and add an OpenCL path for the GPU, and the same runtime compiles for iOS with Metal, for macOS, Linux and Windows, and for browsers through WebAssembly. Alibaba distributes the app as a downloadable APK and warns that the first version was tested only on a OnePlus 13 and a Xiaomi 14 Ultra, so low-spec phones may run slowly or not at all. MNN is published under the Apache 2.0 license. Diagram: Alibaba. The architecture overview from the MNN repository. Source: https://github.com/alibaba/MNN Read the article: https://llmobile.kavents.com/ticker/alibaba-mnn/AlibabaAndroidQuantisationQwenOpen sourcehttps://llmobile.kavents.com/ticker/llama-stack/https://llmobile.kavents.com/ticker/llama-stack/Wed, 25 Sep 2024 19:30:00 +0200Meta announced Llama Stack at Connect on September 25, 2024, a set of REST APIs plus client SDKs for Python, Node, Swift and Kotlin that cover the same Llama features wherever the model runs. The repository defines APIs for Inference, Safety, Memory, Agentic System, Evaluation, Post Training, Synthetic Data Generation and Reward Scoring, and lists PyTorch ExecuTorch as the on-device provider, covering inference and agents on iOS. The models announced at the same event are covered separately. Llama Stack separates providers from distributions. A provider is the implementation behind an API, which can be local code or a pointer to a remote service, and a distribution assembles providers behind one endpoint. Meta’s README states that providers can be mixed, that a hobbyist can serve a small model locally and use a cloud provider for a large one, and that the higher level APIs the app works with do not change, including across the server and mobile-device boundary. Diagram: Meta. From the Llama Stack repository documentation. On iOS the local provider is a Swift class called LocalInference, which the documentation describes as powered by ExecuTorch. It implements the same Inference interface the Swift client uses for remote servers, so an app that already streams chat completions or agent turns keeps its calling code and swaps the service it was given. Developers export the model to an ExecuTorch .pte file, bundle it with tokenizer.model in the app, and link the XNNPACK, Core ML and MPS backends plus the quantized and optimised kernels, with Meta’s own example loading a SpinQuant build of Llama 3.2 1B. The Swift package is added through Xcode’s package manager and generates its types from the Llama Stack OpenAPI spec, and Meta listed its first release on October 10, 2024. The same iOS documentation stated that on-device inference for Android was coming soon, and pointed developers to ExecuTorch’s own Android demo app in the meantime. Llama Stack, the Swift client and the Kotlin client are published under the MIT license. Update, December 6, 2024. Meta released version 0.0.54 of the Kotlin client, which adds on-device inference on Android. The README names ExecuTorch as the local inference distributor, has developers run a script that fetches a prebuilt executorch.aar with the XNNPACK delegate into the app’s libs directory, and shows the only difference between the two modes as the client builder, LlamaStackClientLocalClient with a model and tokenizer path against LlamaStackClientOkHttpClient with a base URL. The chatCompletion call that follows is identical. Source: https://github.com/meta-llama/llama-stack Read the article: https://llmobile.kavents.com/ticker/llama-stack/MetaLlamaiOSAndroidDeveloper toolshttps://llmobile.kavents.com/ticker/torchchat/https://llmobile.kavents.com/ticker/torchchat/Tue, 30 Jul 2024 18:00:00 +0200PyTorch released torchchat on July 30, 2024, a small codebase showing how to run Llama 3 and other language models on laptops, desktops and phones. For the phone path the team reports more than 8 tok/s for Llama 3 8B Instruct on a Samsung Galaxy S23 and on iPhone, using 4-bit GPTQ quantisation through ExecuTorch. Diagram: PyTorch. The repository names iOS 17 or later with at least 8 GB of RAM as its requirement, which it narrows to iPhone 15 Pro and newer or an iPad with Apple silicon, and on Android any device covered by the XNNPACK CPU backend, with 12 GB of RAM and 20 GB of storage recommended. Its model table marks Llama 3.2 1B and 3B, Llama 3.1 8B, Llama 2 7B, Mistral 7B, IBM Granite 3.1 and DeepSeek R1 Distill 8B as mobile friendly. The 11B vision version of Llama 3.2 and the 13B and 70B Llama 2 checkpoints carry no such mark. Getting a model onto a phone is one export step that turns a downloaded checkpoint into a .pte file for ExecuTorch, the PyTorch edge runtime that reached version 1.0 in October 2025. The mobile preset shipped in the repository quantises the embedding table to 4-bit in groups of 32 values and the linear layers to 4-bit weights whose activations are quantised to 8-bit at runtime, in groups of 256. The quantisation guide offers 4-bit and 8-bit embeddings at group sizes from 32 to 256 across all execution modes, lists the dynamic-activation scheme as the supported one for ExecuTorch, and notes that smaller groups preserve accuracy while larger ones run faster. Both demo apps are sample harnesses rather than shipped products. On iOS the repository points at the ExecuTorch LLaMA example project in Xcode, which needs a provisioning profile carrying Apple’s increased memory limit entitlement, after which the model and tokenizer files are copied into the app’s folder on the device. On Android the repository carries its own app skeleton for Android Studio, built against a prebuilt ExecuTorch library, with the model and tokenizer pushed to the phone over adb. The other two paths in torchchat do not touch phones. One runs the model in Python through a chat and generate command line, a browser interface or a local server, the other compiles it ahead of time with PyTorch’s AOTInductor into a binary that a C++ runner loads on a desktop or server. The code is under a BSD 3-Clause license, and PyTorch halted active development on May 19, 2025, saying vLLM had become the dominant option for server-side inference. Source: https://github.com/pytorch/torchchat Read the article: https://llmobile.kavents.com/ticker/torchchat/PyTorchLlamaAndroidiOSQuantisationhttps://llmobile.kavents.com/ticker/executorch-alpha/https://llmobile.kavents.com/ticker/executorch-alpha/Tue, 30 Apr 2024 18:00:00 +0200PyTorch released ExecuTorch alpha on April 30, 2024, the first version of its edge runtime aimed at large language models rather than small vision and speech networks. PyTorch says the work enables Llama 2 7B to run on iPhone 15 Pro, iPhone 15 Pro Max and Samsung Galaxy S22, S23 and S24 phones, with early support for Llama 3 8B. The example project shipped with the release reports 8.15 to 11.6 tok/s for the quantised 7B model, measured by PyTorch over adb on a Galaxy S22, a Galaxy S24 and a OnePlus 12. The 0.1 preview from October 2023 had no such path. Fitting a 7B model on a phone came down to quantisation. Every linear layer carries 4-bit weights in groups of 128 or 256 values, with activations quantised to 8-bit at runtime from their observed range. PyTorch measured WikiText perplexity at 9.16 for the full-precision model against 10.2 and 10.7 for the two group sizes, and notes that groups smaller than 128 were not enabled because the resulting files were still too large, since the embedding table and the weight scales remained in 32-bit floats. Support for 16-bit floats was under way. Arm, Apple and Qualcomm Technologies worked on the release, and PyTorch describes the mobile accelerator paths through Core ML, MPS, TOSA and the Qualcomm AI Stack as ongoing work rather than finished. The release notes add Snapdragon 8 Gen 3 support and 4-bit and 16-bit quantisation on the Qualcomm side, over 100 operators on Apple’s MPS backend, and a Vulkan delegate for mobile GPUs. PyTorch also credits Google for joint work on XNNPACK, the CPU library that carried the language model numbers, and says MediaTek was enabling the Llama models on its own chips. PyTorch was explicit about what alpha did not yet do. The release notes mark XNNPACK as the only finished performance path for generative models and everything else as in progress, and the announcement calls on-device LLM support early while asking the community for help building GPU and NPU delegates and expanding the quantisation schemes. Meta was already shipping the runtime elsewhere, using it for hand tracking on Quest 3, for several models on the Ray-Ban Meta smart glasses and in a beginning rollout inside Instagram. The Python package became installable from PyPI with this release. Source: https://pytorch.org/blog/executorch-alpha/ Read the article: https://llmobile.kavents.com/ticker/executorch-alpha/PyTorchLlamaQuantisationAppleQualcommhttps://llmobile.kavents.com/ticker/mediapipe-llm-inference-api/https://llmobile.kavents.com/ticker/mediapipe-llm-inference-api/Thu, 07 Mar 2024 17:00:00 +0100Google released the MediaPipe LLM Inference API on March 7, 2024, an experimental way to run language models fully on device from web, Android and iOS apps. Four open models are supported at launch, Gemma 2B, Phi 2, Falcon 1B and Stable LM 3B. The API quantises weights to int8, with Gemma 2B using mixed 4-bit and 8-bit weights. Google measured throughput on unnamed high-end devices with a 1024-token input prompt and a maximum of 1280 tokens. In its charts, Gemma 2B at int4 prefills at roughly 680 tokens per second on WebGPU and on an Android GPU, and decodes at about 57 tokens per second on WebGPU against 31 on an Android GPU and 27 on iOS. Chart: Google. Chart: Google. Gemma 2B at int4 was the only model that ran on iOS. Google marks the Android version as intended for experimental and research use only, and points production apps to the Gemini API or to Gemini Nano through Android AICore instead. On iOS, Gemma 2B at int4 was the only model the team could run, which Google attributes to the memory available on the platform. Source: https://developers.googleblog.com/en/large-language-models-on-device-with-mediapipe-and-tensorflow-lite/ Read the article: https://llmobile.kavents.com/ticker/mediapipe-llm-inference-api/GoogleMediaPipeDeveloper toolsAndroidiOShttps://llmobile.kavents.com/ticker/llmfarm/https://llmobile.kavents.com/ticker/llmfarm/Wed, 14 Jun 2023 18:14:00 +0200Artem Savkin published the first release of LLMFarm on June 14, 2023, an iOS and macOS app that loads a quantised model file kept on the device and holds a chat with it offline. The app is built on ggml and llama.cpp, and the README asks for iOS 16 or later and macOS 13 or later. Screenshot from the first release. Screenshot: LLMFarm. The app reads two llama.cpp file formats, the older ggjtv3 files with a .bin extension and GGUF files, according to the project FAQ. The first release covered LLaMA, GPT-NeoX and GPT-2 style models, with 3B builds of OpenLLaMa, StableLM and Dolly v2 among those it listed as working on an iPhone. The README has since added Gemma, Phi, Qwen, Mixtral, RWKV and LLaVA-style vision models, and marks most of them as running through Metal, Apple’s GPU interface, which it says does not work on Intel Macs. Quantisation happens before a model reaches the phone, and the project’s on-device notes list 7B Llama 2 and Orca builds at 3-bit and 4-bit as the ones tried on an iPhone 12 Pro Max, with no published token rates. Each chat writes a JSON file of inference options, which the first release documented with defaults of temperature 0.8, top-k 40, top-p 0.95, a 2048-token context and a repeat penalty of 1.1. The app also offers tail free sampling, locally typical sampling, Mirostat, greedy decoding and grammar-constrained output. Models arrive either through an add model button that imports a file from local storage or through a download list inside the app, and the sandboxed app keeps them in its own models directory. Memory sets the ceiling on an iPhone. The first release’s model table marked 3B models as working on an iPhone 12 Pro Max and 7B and 13B models as returning a “bad alloc” error, and the README said extended memory might be needed to solve it. The FAQ states that the Extended Virtual Addressing and increased-memory-limit entitlements are required to run models larger than 3B, and that the most common reason for a crash is lack of memory. LLMFarm is published under the MIT license, and the README now says the app is temporarily unavailable in TestFlight and the App Store. Source: https://github.com/guinmoon/LLMFarm Read the article: https://llmobile.kavents.com/ticker/llmfarm/llama.cppiOSiPhoneOpen sourceQuantisationhttps://llmobile.kavents.com/ticker/mlc-llm-consumer-devices/https://llmobile.kavents.com/ticker/mlc-llm-consumer-devices/Mon, 01 May 2023 18:00:00 +0200The MLC project released MLC LLM on May 1, 2023, a compiler and runtime that deploys language models natively on phones, browsers and consumer hardware rather than serving them from a data centre. The team pointed users to a TestFlight build to try the chat app on iOS. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: MLC. The iOS app generating a reply on the phone. Open the MLC post MLC LLM compiles models through the Apache TVM Unity stack and maps them to the Vulkan and Metal graphics APIs, which the team says covers the majority of consumer platforms. Targets named at launch include Windows, Linux, macOS and iOS, plus AMD, NVIDIA, Intel and Apple GPUs and a Steam Deck. A companion project, WebLLM, offloads the same models into web browsers through WebGPU. Three weeks later the team added RedPajama-3B across the same set of targets, with a browser demo and a command line tool for desktop machines. The code is on GitHub under the Apache 2.0 license. The project now exposes an OpenAI-compatible API through a REST server as well as Python, JavaScript, iOS and Android bindings, and covers Android GPUs via OpenCL alongside CUDA and ROCm on the desktop. Source: https://blog.mlc.ai/2023/05/01/bringing-accelerated-llm-to-consumer-hardware Read the article: https://llmobile.kavents.com/ticker/mlc-llm-consumer-devices/MLC LLMiOSiPhoneOpen sourceWebGPUhttps://llmobile.kavents.com/ticker/sherpa/https://llmobile.kavents.com/ticker/sherpa/Mon, 27 Mar 2023 20:46:00 +0200Bip-Rep published Sherpa on March 27, 2023, a Flutter app that runs llama.cpp on the phone itself, so the chat works with no network connection. The README asks users to bring their own weights, Meta’s 7B LLaMA converted to a ggml model file, which the app loads through a file picker, and version 2.1.0 added Vicuna support and macOS builds next to the Android and Windows ones. The project publishes no tokens per second figure and states no minimum memory, and the only device it names is a OnePlus 7 with 8 GB of RAM in the demo video below, which the README says was recorded at real speed. The code is MIT licensed. Development stopped three weeks after it started. All 43 commits fall between March 27 and April 14, 2023, the last of the seven releases went out on April 14, 2023, and the llama.cpp fork the app builds against was last touched on April 12, 2023, four months before llama.cpp moved to the GGUF format. Sherpa reads the older ggml files rather than the GGUF models that have been standard since, and requests for GGUF support have gone unanswered, with 23 of the 29 issues ever filed still open. A contributor answered one of them by writing that the original developers had abandoned the project, and pointed to his own fork, Maid. ▶A working chatGPT running on a mobile using llama.cppLoading connects your browser to www.youtube-nocookie.com, which may process your IP address and use cookies. Load contentOpen externallyVideo: tibzejoker. Source: https://github.com/Bip-Rep/sherpa Read the article: https://llmobile.kavents.com/ticker/sherpa/llama.cppAndroidLlamaOpen source