https://llmobile.kavents.com/tags/ios/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/airgap/https://llmobile.kavents.com/ticker/airgap/Wed, 29 Apr 2026 08:06:11 +0200Xavier Puspus published Airgap on April 29, 2026, a React Native starter kit for branded customer support apps that have to keep answering when the network is slow, missing or deliberately blocked. Its configuration file pins a Gemma 4 E2B instruction-tuned model in Q3_K_S GGUF form, about 2.4 GB, which the app downloads on first launch and runs through llama.rn, the React Native binding for llama.cpp. The README states that Airgap searches bundled local documents first and uses a model only to phrase what that retrieval returned, and it lists Apple Foundation Models and Google’s ML Kit GenAI Prompt API as two further on-device answer providers an operator can put ahead of the downloaded file. The repository ships seven example configurations, for airline, banking, electric utility, healthcare, insurance, telecom and water utility support, each with its own branding, prompts and JSON knowledge files, and the downloaded-model path targets iOS 15.1 and newer and Android API 24 and newer. Airgap is published under the MIT license. It is an early single-author project, with 54 commits and no tagged release, and almost all of the code arrived in one squashed initial commit followed by a burst of work in August 2026. The README lists physical-device evaluation of the Apple and Android system providers as an open release gate and labels the published demo recordings as Android emulator and iOS simulator footage rather than evidence of either system model running on a phone. Source: https://github.com/xmpuspus/airgap Read the article: https://llmobile.kavents.com/ticker/airgap/Open sourceAndroidiOSGemmallama.cpphttps://llmobile.kavents.com/ticker/off-grid-offline-ai-app/https://llmobile.kavents.com/ticker/off-grid-offline-ai-app/Sat, 14 Feb 2026 23:39:00 +0100Ali Chherawalla has posted Off Grid to Show HN, an Android and iOS app that runs text generation, image generation, vision and speech recognition entirely on the device. The submission reached 124 points and 66 comments. The repository is licensed under MIT. The app bundles llama.cpp for text, Stable Diffusion for images and whisper.cpp for speech to text. It accepts any GGUF file and lists Qwen 3, Llama 3.2, Gemma 3 and Phi-4 among supported models; vision runs on SmolVLM, Qwen3-VL or Gemma 3n. It also connects to OpenAI-compatible servers on a local network, and builds a project knowledge base from PDFs chunked and embedded on-device with a bundled MiniLM model into SQLite. The README reports 15 to 30 tokens per second on flagship CPUs and 20 to 40 tokens per second through Adreno GPUs via OpenCL on a Snapdragon 8 Gen 2 or newer. Image generation is given as 5 to 10 seconds on the Hexagon NPU and vision inference at about 7 seconds. The NPU path is marked experimental: it accelerates only Q4_0 and Q8_0 quantisation, and K-quants fall back to the CPU. Comments disputed whether models of that size are usable. One reader wrote: Ive tried a lot of self hosted models at a lot of sizes; those small models are not good enough, and do not have a context long enough to be useful for most everyday operations. Chherawalla replied that “you don’t need claude level performance for a lot of day to day tasks” and that Qwen3 8B, or 4B quantised, is “actually quite good”. Other threads compared the app with PocketPal AI, SmolChat and Google’s AI Edge Gallery, asked for an F-Droid build, and reported that image generation required network access on GrapheneOS. A reader on a Samsung device reported the interface did not move for the on-screen keyboard; the developer shipped a fix the same day. Source: https://news.ycombinator.com/item?id=47019133 Read the article: https://llmobile.kavents.com/ticker/off-grid-offline-ai-app/AndroidiOSOpen sourcellama.cppImage generationhttps://llmobile.kavents.com/ticker/apple-foundation-models-framework/https://llmobile.kavents.com/ticker/apple-foundation-models-framework/Mon, 09 Jun 2025 21:00:00 +0200Apple opened its on-device foundation model to third-party apps at WWDC25 on June 9, 2025. The Foundation Models framework gives any app direct access to the roughly 3-billion-parameter model, which runs offline and costs developers nothing per call. Developers annotate Swift data structures with the @Generable macro, and the framework applies constrained decoding, which Apple calls guided generation, so the model returns those structures rather than free text. Tool calling lets the model invoke functions the app supplies. Developers who need more control can train rank 32 adapters with a Python toolkit. Apple compressed the model to 2 bits per weight using quantisation-aware training. It splits the transformer into two blocks at a 5:3 depth ratio and shares the key-value cache between them, which Apple says cuts KV cache memory use by 37.5 percent. Images are handled by a 300M-parameter ViTDet-L vision encoder. The model is designed to support 15 languages. The companion server model on Private Cloud Compute uses what Apple calls a Parallel-Track Mixture-of-Experts design, where independent transformer tracks cut how often the tracks have to synchronise with each other. Diagram: Apple. The server model, not the on-device one. Source: https://machinelearning.apple.com/research/apple-foundation-models-2025-updates Read the article: https://llmobile.kavents.com/ticker/apple-foundation-models-framework/AppleiOSDeveloper toolsQuantisationAgentshttps://llmobile.kavents.com/ticker/locally-ai/https://llmobile.kavents.com/ticker/locally-ai/Sun, 13 Apr 2025 09:00:00 +0200Adrien Grondin released Locally AI on April 13, 2025, an iPhone and iPad app that downloads open-weight language models and runs them entirely on the device. The App Store listing gives that release date and lists the app as free with no in-app purchases. Once a model is on disk the app needs no account and no internet connection, and Grondin states that no data is collected and nothing is sent to a cloud service. The site at launch named Meta Llama 3.2 and Llama 3.1, Google Gemma 2, Qwen 2 and DeepSeek R1 as the models on offer, and described them as language and vision models, so a picture can go into the prompt alongside text. A user picks a model, downloads it once and can edit the system prompt the model runs under. Grondin builds the app on MLX, Apple’s machine learning framework for Apple silicon, and attributes the performance to unified memory, where the CPU and the GPU reach the same memory pool so arrays need not be copied between them. The app fetches open-weight model files itself rather than calling any model that ships with the operating system. Neither the site nor the App Store entry names a minimum chip. The site says only that Locally AI covers recent iPhone and iPad models and is optimised for Apple silicon, and the listing asks for iOS 18.1 or iPadOS 18.1 or later. No generation speed, memory requirement or model file size is given anywhere in the developer’s own material, and the launch site was a bought template whose team, testimonial and FAQ sections still carried placeholder Latin text. Update, April 8, 2026. Grondin added support for Apple’s on-device Foundation Models framework in September 2025, so the app can open a conversation with the model built into iOS 26 without downloading anything, which Simon Willison recorded on September 21, 2025. A Mac version followed and requires macOS 26 or later. LM Studio announced on April 8, 2026 that Locally AI had joined the company and that Grondin had moved with it, and the App Store entry now reads Locally AI by LM Studio with Element Labs Inc as the seller. Source: https://locallyai.app/ Read the article: https://llmobile.kavents.com/ticker/locally-ai/AppleApple SiliconMLXiOSiPhonehttps://llmobile.kavents.com/ticker/huggingsnap/https://llmobile.kavents.com/ticker/huggingsnap/Wed, 19 Mar 2025 16:16:00 +0100Hugging Face published HuggingSnap, an iPhone app that describes what the camera is pointed at, with the model running on the phone rather than in the cloud. The app’s runtime configuration loads SmolVLM2 at 500M parameters in Apple’s MLX format. Hugging Face had trailed the app in its SmolVLM2 announcement on February 20, 2025, writing that the 500M model lets users analyse and understand video content directly on their device with no cloud required. The README describes the app as a way to point the camera and have text translated or summarised, identify plants and animals and learn more about surrounding objects. Inference runs through MLX Swift, Apple’s machine learning framework for Apple silicon, with a modified fork of Apple’s MLX example code supplying the vision-language support. The app pulls the weights from the Hugging Face Hub the first time it runs, then holds Metal memory to 82 percent of what iOS reports as available to the process. Input comes from the live camera, from stills and clips captured in the app, and from photos and videos picked out of the library. A tap on the shutter takes a picture and a press and hold records video, after which the app offers a describe action for images and a summarise action for video, with questions typed or dictated. The app’s own interface states that photos and videos are processed entirely on the device and that no data is sent to the cloud. The README states that installation needs an iPhone running iOS 18, and the Xcode project sets iOS 18.0 as the deployment target and iPhone as the only device family, with no Mac Catalyst or visionOS build. TechCrunch reported the app as a free App Store download on March 19, 2025, quoting the listing as saying it works offline and processes all data on the phone. The SmolVLM2 weights are Apache 2.0, while the app repository itself carries no licence file. Source: https://github.com/huggingface/HuggingSnap Read the article: https://llmobile.kavents.com/ticker/huggingsnap/Hugging FaceiOSiPhoneApple SiliconOpen weightshttps://llmobile.kavents.com/ticker/llama-stack/https://llmobile.kavents.com/ticker/llama-stack/Wed, 25 Sep 2024 19:30:00 +0200Meta announced Llama Stack at Connect on September 25, 2024, a set of REST APIs plus client SDKs for Python, Node, Swift and Kotlin that cover the same Llama features wherever the model runs. The repository defines APIs for Inference, Safety, Memory, Agentic System, Evaluation, Post Training, Synthetic Data Generation and Reward Scoring, and lists PyTorch ExecuTorch as the on-device provider, covering inference and agents on iOS. The models announced at the same event are covered separately. Llama Stack separates providers from distributions. A provider is the implementation behind an API, which can be local code or a pointer to a remote service, and a distribution assembles providers behind one endpoint. Meta’s README states that providers can be mixed, that a hobbyist can serve a small model locally and use a cloud provider for a large one, and that the higher level APIs the app works with do not change, including across the server and mobile-device boundary. Diagram: Meta. From the Llama Stack repository documentation. On iOS the local provider is a Swift class called LocalInference, which the documentation describes as powered by ExecuTorch. It implements the same Inference interface the Swift client uses for remote servers, so an app that already streams chat completions or agent turns keeps its calling code and swaps the service it was given. Developers export the model to an ExecuTorch .pte file, bundle it with tokenizer.model in the app, and link the XNNPACK, Core ML and MPS backends plus the quantized and optimised kernels, with Meta’s own example loading a SpinQuant build of Llama 3.2 1B. The Swift package is added through Xcode’s package manager and generates its types from the Llama Stack OpenAPI spec, and Meta listed its first release on October 10, 2024. The same iOS documentation stated that on-device inference for Android was coming soon, and pointed developers to ExecuTorch’s own Android demo app in the meantime. Llama Stack, the Swift client and the Kotlin client are published under the MIT license. Update, December 6, 2024. Meta released version 0.0.54 of the Kotlin client, which adds on-device inference on Android. The README names ExecuTorch as the local inference distributor, has developers run a script that fetches a prebuilt executorch.aar with the XNNPACK delegate into the app’s libs directory, and shows the only difference between the two modes as the client builder, LlamaStackClientLocalClient with a model and tokenizer path against LlamaStackClientOkHttpClient with a base URL. The chatCompletion call that follows is identical. Source: https://github.com/meta-llama/llama-stack Read the article: https://llmobile.kavents.com/ticker/llama-stack/MetaLlamaiOSAndroidDeveloper toolshttps://llmobile.kavents.com/ticker/olmoe/https://llmobile.kavents.com/ticker/olmoe/Tue, 03 Sep 2024 19:08:20 +0200Ai2 published OLMoE-1B-7B on September 3, 2024, a language model with 6.9B total parameters of which only 1.3B run on any given token. It is a sparse mixture of experts, meaning each layer holds 64 small feedforward networks and a small router picks 8 of them per token, so the rest of the model stays idle. Compute per token lands near that of a dense 1B model, while all 6.9B parameters still have to be held in memory. Ai2 pretrained the model on 5 trillion tokens and reports in the paper that it leads every open model in its active-parameter class. On MMLU the paper puts OLMoE-1B-7B at 54.1, against 48.5 for DCLM-1B, 33.6 for TinyLlama-1B and 32.1 for Ai2’s own dense OLMo-1B. The paper states the model matches or outperforms the dense OLMo-7B at the end of pretraining while using less than half the training FLOPs, and that it beats Llama2-7B on MMLU, scored at 46.2, despite needing roughly 6 to 7 times less compute per forward pass. The instruction-tuned version is reported above Llama2-13B-Chat and DeepSeekMoE-16B. For the training-cost claim the authors ran a controlled comparison, training a 1.3B dense model and a 1.3B active, 6.9B total mixture of experts on 128 H100 GPUs for 130B tokens each. Ai2 reports the sparse model reaching the dense model’s final quality with about 3 times fewer tokens and FLOPs, but only about 2 times faster in wall-clock time, because the larger weight set dropped throughput to 23,600 tokens per second per GPU against 37,500 for the dense run. The full pretraining run used 256 H100 GPUs for around 10 days. Ai2 shipped an iOS app on February 11, 2025 that keeps the whole model on the device, so prompts and responses never reach a server and generation keeps working in airplane mode. Ai2 names the iPhone 15 Pro and 15 Pro Max, the four iPhone 16 models and M-series iPads going back to the 2021 iPad Pro, and states that the first version needs an iOS device with 8 GB of memory. The app runs a 4-bit quantised build of the January 2025 instruction-tuned model, which Ai2 measures at 41 tokens per second on average on an iPhone 16 Pro, on a stack Ai2 built with GenUI on top of llama.cpp. Weights, the training data mix, the training code and the run logs are all published under Apache 2.0, together with 244 intermediate checkpoints. The comparison table in the paper lists OLMoE as the only mixture-of-experts release that opens all four, with most of the other models in the table offering a single checkpoint and no logs. The app is in the App Store and its source code is on GitHub. Screenshot: Ai2. Update, April 14, 2025. Ai2 kept working on the app after the February launch and added macOS support on March 7, 2025 through Mac Catalyst, Apple’s compatibility layer that runs the same iOS code natively on a Mac, together with a counter that shows how many tokens a reply used. A later version reworked the panel that reports generation metrics, and the most recent release, from April 14, 2025, renamed the app to Ai2 OLMoE. Nothing has shipped in the repository since. Source: https://arxiv.org/abs/2409.02060 Read the article: https://llmobile.kavents.com/ticker/olmoe/Ai2Mixture of expertsiOSOpen weightsResearchhttps://llmobile.kavents.com/ticker/torchchat/https://llmobile.kavents.com/ticker/torchchat/Tue, 30 Jul 2024 18:00:00 +0200PyTorch released torchchat on July 30, 2024, a small codebase showing how to run Llama 3 and other language models on laptops, desktops and phones. For the phone path the team reports more than 8 tok/s for Llama 3 8B Instruct on a Samsung Galaxy S23 and on iPhone, using 4-bit GPTQ quantisation through ExecuTorch. Diagram: PyTorch. The repository names iOS 17 or later with at least 8 GB of RAM as its requirement, which it narrows to iPhone 15 Pro and newer or an iPad with Apple silicon, and on Android any device covered by the XNNPACK CPU backend, with 12 GB of RAM and 20 GB of storage recommended. Its model table marks Llama 3.2 1B and 3B, Llama 3.1 8B, Llama 2 7B, Mistral 7B, IBM Granite 3.1 and DeepSeek R1 Distill 8B as mobile friendly. The 11B vision version of Llama 3.2 and the 13B and 70B Llama 2 checkpoints carry no such mark. Getting a model onto a phone is one export step that turns a downloaded checkpoint into a .pte file for ExecuTorch, the PyTorch edge runtime that reached version 1.0 in October 2025. The mobile preset shipped in the repository quantises the embedding table to 4-bit in groups of 32 values and the linear layers to 4-bit weights whose activations are quantised to 8-bit at runtime, in groups of 256. The quantisation guide offers 4-bit and 8-bit embeddings at group sizes from 32 to 256 across all execution modes, lists the dynamic-activation scheme as the supported one for ExecuTorch, and notes that smaller groups preserve accuracy while larger ones run faster. Both demo apps are sample harnesses rather than shipped products. On iOS the repository points at the ExecuTorch LLaMA example project in Xcode, which needs a provisioning profile carrying Apple’s increased memory limit entitlement, after which the model and tokenizer files are copied into the app’s folder on the device. On Android the repository carries its own app skeleton for Android Studio, built against a prebuilt ExecuTorch library, with the model and tokenizer pushed to the phone over adb. The other two paths in torchchat do not touch phones. One runs the model in Python through a chat and generate command line, a browser interface or a local server, the other compiles it ahead of time with PyTorch’s AOTInductor into a binary that a C++ runner loads on a desktop or server. The code is under a BSD 3-Clause license, and PyTorch halted active development on May 19, 2025, saying vLLM had become the dominant option for server-side inference. Source: https://github.com/pytorch/torchchat Read the article: https://llmobile.kavents.com/ticker/torchchat/PyTorchLlamaAndroidiOSQuantisationhttps://llmobile.kavents.com/ticker/apple-intelligence-foundation-models/https://llmobile.kavents.com/ticker/apple-intelligence-foundation-models/Mon, 10 Jun 2024 21:00:00 +0200Apple introduced Apple Intelligence on June 10, 2024, built on a roughly 3-billion-parameter language model that runs on the device and a larger model that runs on Apple silicon servers through Private Cloud Compute. The on-device model uses a 49K token vocabulary, the server model 100K. Apple compresses the on-device model with a mix of 2-bit and 4-bit weights, averaging 3.7 bits per weight, and says an option at 3.5 bits per weight costs no significant quality. On an iPhone 15 Pro the company reports a time to first token of about 0.6 milliseconds per prompt token and 30 tokens per second, both measured before token speculation. Rather than ship a separate model per feature, Apple swaps LoRA adapters on top of the shared base model. The adapters are stored in 16 bits and, at rank 16, take tens of megabytes each. Apple says human graders preferred its models over comparable ones including GPT-3.5, GPT-4, Llama, Mistral and Gemma across its evaluations. Diagram: Apple. Source: https://machinelearning.apple.com/research/introducing-apple-foundation-models Read the article: https://llmobile.kavents.com/ticker/apple-intelligence-foundation-models/AppleiPhoneiOSQuantisationApple Siliconhttps://llmobile.kavents.com/ticker/mediapipe-llm-inference-api/https://llmobile.kavents.com/ticker/mediapipe-llm-inference-api/Thu, 07 Mar 2024 17:00:00 +0100Google released the MediaPipe LLM Inference API on March 7, 2024, an experimental way to run language models fully on device from web, Android and iOS apps. Four open models are supported at launch, Gemma 2B, Phi 2, Falcon 1B and Stable LM 3B. The API quantises weights to int8, with Gemma 2B using mixed 4-bit and 8-bit weights. Google measured throughput on unnamed high-end devices with a 1024-token input prompt and a maximum of 1280 tokens. In its charts, Gemma 2B at int4 prefills at roughly 680 tokens per second on WebGPU and on an Android GPU, and decodes at about 57 tokens per second on WebGPU against 31 on an Android GPU and 27 on iOS. Chart: Google. Chart: Google. Gemma 2B at int4 was the only model that ran on iOS. Google marks the Android version as intended for experimental and research use only, and points production apps to the Gemini API or to Gemini Nano through Android AICore instead. On iOS, Gemma 2B at int4 was the only model the team could run, which Google attributes to the memory available on the platform. Source: https://developers.googleblog.com/en/large-language-models-on-device-with-mediapipe-and-tensorflow-lite/ Read the article: https://llmobile.kavents.com/ticker/mediapipe-llm-inference-api/GoogleMediaPipeDeveloper toolsAndroidiOShttps://llmobile.kavents.com/ticker/apple-mlx/https://llmobile.kavents.com/ticker/apple-mlx/Tue, 05 Dec 2023 01:04:00 +0100Apple machine learning research shipped the first public release of MLX on December 5, 2023, an array framework for machine learning on Apple silicon. The repository carries a v0.0.2 tag dated that day and the same version went up on PyPI minutes later, after an initial commit on November 28. Arrays in MLX live in shared memory, so an operation can run on the CPU or on the GPU without the data being copied from one place to the other. That behaviour comes from the hardware. Apple silicon uses a unified memory architecture in which the CPU and the GPU have direct access to the same memory pool, and the MLX documentation states that an array therefore has no device of its own. Code names the device when it runs an operation instead of moving an array to a device first, and when two operations on different devices depend on each other, the documentation says the MLX scheduler inserts the dependency between the streams automatically. Computation is lazy. The documentation states that operations only record a compute graph and that nothing is computed until an eval call asks for a result, which is what lets MLX apply function transformations such as automatic differentiation and automatic vectorisation to the graph. It gives memory as the second reason, since a model whose weights are created as float32 and then replaced with float16 before any evaluation peaks at half the memory eager computation would need. The MLX Swift bindings followed a week later, with an initial commit on December 12, 2023, and the package manifest lists macOS 14, iOS 17, tvOS 17 and visionOS 1 as supported platforms. The example apps build for iOS as well as macOS and include a chat client for language and vision-language models, a text generation demo that downloads weights from Hugging Face, and Stable Diffusion image generation. The LLMEval example uses the Increased Memory Limit entitlement on iOS, which its README attributes to the size of language model weights, and caps the MLX buffer cache at 20 MB. MLX LM is the Python package that runs and fine-tunes language models on top of MLX, first published to PyPI on January 12, 2024. It installs the mlx_lm.generate and mlx_lm.chat command line tools, whose default model is a 4-bit quantised Llama 3.2 3B Instruct from the mlx-community organisation on Hugging Face, and it can quantise and upload converted models back to the Hub. MLX is published under the MIT license, and the install documentation lists Apple silicon, macOS 14.0 or newer and Python 3.10 or newer for the macOS package, alongside CUDA and CPU-only builds for Linux added later. Source: https://github.com/ml-explore/mlx Read the article: https://llmobile.kavents.com/ticker/apple-mlx/AppleApple SiliconDeveloper toolsOpen sourceiOShttps://llmobile.kavents.com/ticker/llm-swift/https://llmobile.kavents.com/ticker/llm-swift/Mon, 04 Dec 2023 00:11:00 +0100The South Korean developer eastriverlee published the first release of LLM.swift late on December 3, 2023, a Swift package that wraps llama.cpp so an iOS or Mac app can load a GGUF model file and generate text on the device in a few lines. The launch example creates a model object from a file bundled with the app and awaits a reply that streams back token by token into a SwiftUI view. The package declared macOS 12, iOS 14, watchOS 4, tvOS 14 and visionOS 1 as its minimum versions, and required Swift 5.9. Screenshot: LLM.swift. The test app shipped in the first release README. Sitting on llama.cpp means the package reads GGUF files and nothing else, and the project states that any model llama.cpp runs should run here too, which is a different route from Hugging Face’s swift-transformers and its Core ML pipeline. Chat formatting was left to the app author at first, a closure that assembled the prompt plus a stop string that ends generation, and the release seven weeks later replaced that with ready-made templates for ChatML, Alpaca and Llama, with Mistral and Gemma added since. A developer adds the package through the Swift Package Manager by pointing at the repository, and the README recommends models of 3B parameters or fewer on mobile devices, reporting a Mistral 7B model running on an iPad Air 5th generation at 5-bit quantisation and on an iPhone 12 mini at 2-bit. The project has since put out 41 further releases, the most recent in July 2026, and version 3 made the chat template embedded in the GGUF file the default, added function calling and schema-constrained structured output, and switched to a prebuilt llama.cpp framework fetched from an upstream release by URL and checksum. The minimum versions have risen to iOS 16, macOS 13, watchOS 9, tvOS 16 and visionOS 1. Almost all of the roughly 150 commits are eastriverlee’s own, and the package is published under the MIT license. Source: https://github.com/eastriverlee/LLM.swift Read the article: https://llmobile.kavents.com/ticker/llm-swift/Developer toolsiOSllama.cppOpen sourceApple Siliconhttps://llmobile.kavents.com/ticker/hf-swift-transformers/https://llmobile.kavents.com/ticker/hf-swift-transformers/Tue, 08 Aug 2023 16:15:00 +0200Hugging Face published swift-transformers on August 8, 2023, the date of the repository’s first tag, a Swift package that gives iOS and Mac developers a transformers-like API for running language models on the device through Core ML. Pedro Cuenca introduced it as an in-development library built around four modules, Tokenizers, Hub, Generation and Models. The Tokenizers module ports the normalisation and pre-processing rules that the Python tokenizers library keeps as configuration files next to a model on the Hub, so an app does not hardcode those decisions per model and can swap one checkpoint for another. It covered BPE at release, the family used by the GPT models, Falcon and Llama, with Unigram and WordPiece listed as later work. The Hub module downloads those configuration files and the model metadata, and Generation implements greedy decoding and top-k sampling. The Models module wraps a converted Core ML package behind a LanguageModel type, and Core ML then decides whether to run it on the CPU, the GPU or the Neural Engine. Cuenca writes that flexible input shapes run on the CPU only, which is why Hugging Face’s exporters conversion tool switched to fixed sequence lengths, and that the Core ML path cached no attention keys and values at that point, both of which cap how long a context an app can work with. Hugging Face states it tested the package with Llama 2, Falcon, StarCoder and the GPT family including GPT2, distilgpt, GPT-NeoX and GPT-J, and that encoder-decoder models such as T5 and Flan were not supported. The release came alongside swift-chat, a Mac app that loads a Core ML model and doubles as a model tester, plus converted weights for Llama 2 7B and Falcon 7B on the Hub. The package is licensed under Apache 2.0. Screenshot: Hugging Face. The demo app exposes the generation parameters and reports throughput. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: Hugging Face. swift-chat generating a haiku from Llama 2 7B on an M1 MacBook Pro. Open the Hugging Face post Source: https://github.com/huggingface/swift-transformers Read the article: https://llmobile.kavents.com/ticker/hf-swift-transformers/Developer toolsiOSApple SiliconOpen sourceLlamahttps://llmobile.kavents.com/ticker/llmfarm/https://llmobile.kavents.com/ticker/llmfarm/Wed, 14 Jun 2023 18:14:00 +0200Artem Savkin published the first release of LLMFarm on June 14, 2023, an iOS and macOS app that loads a quantised model file kept on the device and holds a chat with it offline. The app is built on ggml and llama.cpp, and the README asks for iOS 16 or later and macOS 13 or later. Screenshot from the first release. Screenshot: LLMFarm. The app reads two llama.cpp file formats, the older ggjtv3 files with a .bin extension and GGUF files, according to the project FAQ. The first release covered LLaMA, GPT-NeoX and GPT-2 style models, with 3B builds of OpenLLaMa, StableLM and Dolly v2 among those it listed as working on an iPhone. The README has since added Gemma, Phi, Qwen, Mixtral, RWKV and LLaVA-style vision models, and marks most of them as running through Metal, Apple’s GPU interface, which it says does not work on Intel Macs. Quantisation happens before a model reaches the phone, and the project’s on-device notes list 7B Llama 2 and Orca builds at 3-bit and 4-bit as the ones tried on an iPhone 12 Pro Max, with no published token rates. Each chat writes a JSON file of inference options, which the first release documented with defaults of temperature 0.8, top-k 40, top-p 0.95, a 2048-token context and a repeat penalty of 1.1. The app also offers tail free sampling, locally typical sampling, Mirostat, greedy decoding and grammar-constrained output. Models arrive either through an add model button that imports a file from local storage or through a download list inside the app, and the sandboxed app keeps them in its own models directory. Memory sets the ceiling on an iPhone. The first release’s model table marked 3B models as working on an iPhone 12 Pro Max and 7B and 13B models as returning a “bad alloc” error, and the README said extended memory might be needed to solve it. The FAQ states that the Extended Virtual Addressing and increased-memory-limit entitlements are required to run models larger than 3B, and that the most common reason for a crash is lack of memory. LLMFarm is published under the MIT license, and the README now says the app is temporarily unavailable in TestFlight and the App Store. Source: https://github.com/guinmoon/LLMFarm Read the article: https://llmobile.kavents.com/ticker/llmfarm/llama.cppiOSiPhoneOpen sourceQuantisationhttps://llmobile.kavents.com/ticker/mlc-llm-consumer-devices/https://llmobile.kavents.com/ticker/mlc-llm-consumer-devices/Mon, 01 May 2023 18:00:00 +0200The MLC project released MLC LLM on May 1, 2023, a compiler and runtime that deploys language models natively on phones, browsers and consumer hardware rather than serving them from a data centre. The team pointed users to a TestFlight build to try the chat app on iOS. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: MLC. The iOS app generating a reply on the phone. Open the MLC post MLC LLM compiles models through the Apache TVM Unity stack and maps them to the Vulkan and Metal graphics APIs, which the team says covers the majority of consumer platforms. Targets named at launch include Windows, Linux, macOS and iOS, plus AMD, NVIDIA, Intel and Apple GPUs and a Steam Deck. A companion project, WebLLM, offloads the same models into web browsers through WebGPU. Three weeks later the team added RedPajama-3B across the same set of targets, with a browser demo and a command line tool for desktop machines. The code is on GitHub under the Apache 2.0 license. The project now exposes an OpenAI-compatible API through a REST server as well as Python, JavaScript, iOS and Android bindings, and covers Android GPUs via OpenCL alongside CUDA and ROCm on the desktop. Source: https://blog.mlc.ai/2023/05/01/bringing-accelerated-llm-to-consumer-hardware Read the article: https://llmobile.kavents.com/ticker/mlc-llm-consumer-devices/MLC LLMiOSiPhoneOpen sourceWebGPUhttps://llmobile.kavents.com/ticker/onnx-runtime/https://llmobile.kavents.com/ticker/onnx-runtime/Tue, 29 Sep 2020 19:00:00 +0200Microsoft released ONNX Runtime 1.5 on September 29, 2020 with a preview of ORT for Mobile Platforms, described in the release notes as minimising build size for mobile and embedded devices. The same release added a reduced operator kernel build, which compiles the runtime with only the operators the target models actually use, and the ORT model format that those reduced builds require. The format documentation records version 1 of the ORT format as introduced in 1.5. Diagram: ONNX Runtime documentation. The pre-built mobile packages, onnxruntime-android for Java, C and C++ and onnxruntime-c and onnxruntime-objc for iOS, carry the full operator set and load .onnx files directly, while a custom build takes models converted to the .ort format. Microsoft’s mobile guide puts numbers on the difference for version 1.18.0, where a custom build supporting the operators of a ResNet50 model produces an Android AAR of 7,532,309 bytes against 24,415,212 bytes for the pre-built package, and an arm64-v8a libonnxruntime.so of 3,962,832 bytes against 16,276,832 bytes. Accelerators are called execution providers. The NNAPI provider requires Android 8.1 or higher and is recommended on Android 9 or higher, the Core ML provider requires iOS 13 or macOS 10.15 and higher, and the XNNPACK provider covers floating-point CPU work on both platforms. Microsoft’s guidance is to start on the CPU provider for a quantised model and on XNNPACK otherwise, then try NNAPI or Core ML if that misses the performance target. The QNN provider reaches Qualcomm Snapdragon chips on Android and Windows through the Qualcomm AI Engine Direct SDK, with pre-built packages for Windows only and an HTP backend that takes quantised models only. Quantisation in ONNX Runtime is 8-bit linear quantisation, offered as dynamic and static post-training quantisation in two representations, QOperator with dedicated quantised operators and QDQ with QuantizeLinear and DequantizeLinear pairs inserted around the originals. The mobile guide states that moving 32-bit weights to 8 bits cuts model size by roughly a factor of 4. The tooling also does block-wise weight-only quantisation to int4 and uint4 for MatMul and Gather nodes, with RTN, HQQ and GPTQ algorithms, and models with GatherBlockQuantized nodes need ONNX Runtime 1.20 or later to run. Generative models run through the separate onnxruntime-genai package, which implements the generation loop, KV cache management, search and sampling around the runtime. Its support matrix lists Android as supported and iOS as on the roadmap, with CPU, QNN, WebGPU, CUDA, DirectML and OpenVINO among the acceleration options. Microsoft reported in April 2024 that RTN int4 quantisation shrinks Phi-3 Mini enough to run on a Samsung Galaxy S21 at what it called a moderate speed, and recommends the int4_accuracy_level=4 variant on mobile. Both projects are published under the MIT license. Source: https://github.com/microsoft/onnxruntime/releases/tag/v1.5.1 Read the article: https://llmobile.kavents.com/ticker/onnx-runtime/MicrosoftAndroidiOSQuantisationDeveloper tools