Mobile AI news ticker

The South Korean developer eastriverlee published the first release of LLM.swift late on December 3, 2023, a Swift package that wraps llama.cpp so an iOS or Mac app can load a GGUF model file and generate text on the device in a few lines. The launch example creates a model object from a file bundled with the app and awaits a reply that streams back token by token into a SwiftUI view. The package declared macOS 12, iOS 14, watchOS 4, tvOS 14 and visionOS 1 as its minimum versions, and required Swift 5.9.

Sitting on llama.cpp means the package reads GGUF files and nothing else, and the project states that any model llama.cpp runs should run here too, which is a different route from Hugging Face’s swift-transformers and its Core ML pipeline. Chat formatting was left to the app author at first, a closure that assembled the prompt plus a stop string that ends generation, and the release seven weeks later replaced that with ready-made templates for ChatML, Alpaca and Llama, with Mistral and Gemma added since. A developer adds the package through the Swift Package Manager by pointing at the repository, and the README recommends models of 3B parameters or fewer on mobile devices, reporting a Mistral 7B model running on an iPad Air 5th generation at 5-bit quantisation and on an iPhone 12 mini at 2-bit.

The project has since put out 41 further releases, the most recent in July 2026, and version 3 made the chat template embedded in the GGUF file the default, added function calling and schema-constrained structured output, and switched to a prebuilt llama.cpp framework fetched from an upstream release by URL and checksum. The minimum versions have risen to iOS 16, macOS 13, watchOS 9, tvOS 16 and visionOS 1. Almost all of the roughly 150 commits are eastriverlee’s own, and the package is published under the MIT license.

Qualcomm launched the Snapdragon 8 Gen 3 at its Snapdragon Summit in Hawaii on October 24, 2023, calling it its first mobile platform designed with generative AI in mind. On the product page the company states support for generative models with up to 10 billion parameters on device and up to 20 tokens per second for large language models.

Qualcomm puts the Hexagon NPU at 98 percent faster than the previous generation and claims 40 percent better performance per watt for sustained AI inference. It lists the Kryo CPU at 30 percent more performance and 20 percent better power efficiency, and the Adreno GPU at 25 percent faster rendering with 25 percent better efficiency.

Among the workloads Qualcomm names for local execution are Stable Diffusion, virtual assistants and chatbots. Devices were announced by ASUS, Honor, iQOO, MEIZU, Nubia, OnePlus, OPPO, realme, Redmi, RedMagic, Sony, vivo, Xiaomi and ZTE.

Microsoft Research published phi-1.5 on September 11, 2023, a 1.3-billion-parameter model that applies the synthetic textbook data approach of phi-1 to common sense reasoning and general world knowledge rather than Python code. The authors report common sense benchmark results comparable to models ten times the size trained on more than ten times as much data, and describe natural language performance as comparable to models five times larger.

phi-1.5 scores 0.734 on WinoGrande against 0.691 for Llama2-7B, 0.444 on ARC-Challenge against 0.434, and 0.526 on SIQA against 0.480, all measured in Microsoft’s own evaluation pipeline. The gap widens on multi-step reasoning, with 40.2 on GSM8K grade-school maths against 14.6 for Llama2-7B, 34.1 on HumanEval against 12.8, and 37.7 on MBPP against 20.8, which the authors note puts its coding scores at or above Llama-65B. MMLU runs the other way, at 0.376 for phi-1.5 and 0.453 for Llama2-7B.

The training set combines 7B tokens from phi-1 with roughly 20B tokens of newly generated textbook-like text, seeded from 20,000 selected topics spanning science, daily activities and theory of mind. That yields a 30B-token dataset whose only non-synthetic part is 6B tokens of filtered code, and training runs over 150B tokens, 80 percent of them from the new synthetic data. Microsoft puts the cost at 1,500 GPU hours on a single A100-80G, against more than 80,000 hours for Llama-7B.

The same report covers two variants built to test what web data adds. phi-1.5-web-only trains purely on 95B tokens of filtered web text, 88B of it from the Falcon RefinedWeb dataset and 7B of code from The Stack and Stack Overflow, while phi-1.5-web mixes filtered web data, phi-1’s code and the synthetic text in shares of roughly 40, 20 and 40 percent. phi-1.5-web comes out ahead on reasoning at 44.6 on GSM8K, 41.4 on HumanEval and 43.5 on MBPP.

The authors list hallucinations and the potential for toxic and biased generations among the traits phi-1.5 shares with larger models, and write that it “is not immune” despite the absence of web data in its training set. On an 86-prompt probe they wrote themselves, phi-1.5 passed 47 and failed 34, while Llama2-7B and Falcon-7B failed 54 and 50. Microsoft released the raw base model with no instruction fine-tuning and no RLHF, and the weights sit on Hugging Face under the MIT license.

Song Han’s group at MIT pushed the first commit of TinyChatEngine on August 24, 2023, according to the repository’s commit history. The library runs quantised language models in plain C and C++ with no dependency on bulky libraries, and the README names x86 CPUs from Intel and AMD, ARM devices including Apple M1 and M2 Macs and the Raspberry Pi, and Nvidia GPUs through CUDA. The lab’s own slides put LLaMA2-7B at 71 ms per token on an M1 Pro and 210 ms per token on an Intel i7-9750H laptop CPU.

The compression behind it comes from two papers by the same lab. AWQ, for activation-aware weight quantisation, was submitted on June 1, 2023 and works from the finding that protecting only 1 percent of the weights, picked by looking at the activation distribution rather than the weights themselves, keeps most of the accuracy at 4 bits. SmoothQuant, submitted on November 18, 2022, handles 8-bit weights and 8-bit activations by shifting the hard part of the job from activations to weights through a mathematically equivalent transformation, and its authors report up to 1.56x speedup and 2x memory reduction.

The backend table lists W4A32, W4A8 and W8A8 on both x86 and ARM CPUs, W4A16 on Nvidia GPUs, and FP32 on CPU. TinyChatEngine reorders the 4-bit weights offline during model conversion rather than at runtime, in a layout tailored to each instruction set, with QM_ARM built for 128-bit SIMD and QM_x86 for 256-bit SIMD. The slides state that expanding the 4-bit weights to int8 and using int8 SIMD multiply-accumulate instead of FP32 gives 1.3x on the Intel i7-9750H and 3x on the M1 Pro.

For the device comparison the lab measured LLaMA2-7B against PyTorch and llama.cpp. On the Intel i7-9750H it reports 210 ms per token for TinyChatEngine, 233 for llama.cpp and 1375 for PyTorch, which it labels a 6.5x gap. On an Nvidia Jetson Orin the figures are 108, 133 and 141 ms per token, and on the M1 Pro TinyChatEngine lands at 71 ms per token against 66 for llama.cpp, with PyTorch running out of memory. The lab publishes no Raspberry Pi timing.

The model zoo ships LLaMA-7B, LLaMA2-7B-chat and LLaMA2-13B-chat plus OPT at 125M, 1.3B and 6.7B, in FP32, AWQ int4 and SmoothQuant int8 variants, with a download script that takes a per-device quantisation flag. macOS users install boost and llvm through Homebrew, and Xcode for the Metal compiler that enables Apple GPU support. The repository carries an MIT license, added the day after the first commit. For the Raspberry Pi the project recommends a board with 8 GB of RAM and says its testing ran on a Raspberry Pi 4 Model B Rev 1.4 with aarch64.

Hugging Face published swift-transformers on August 8, 2023, the date of the repository’s first tag, a Swift package that gives iOS and Mac developers a transformers-like API for running language models on the device through Core ML. Pedro Cuenca introduced it as an in-development library built around four modules, Tokenizers, Hub, Generation and Models.

The Tokenizers module ports the normalisation and pre-processing rules that the Python tokenizers library keeps as configuration files next to a model on the Hub, so an app does not hardcode those decisions per model and can swap one checkpoint for another. It covered BPE at release, the family used by the GPT models, Falcon and Llama, with Unigram and WordPiece listed as later work. The Hub module downloads those configuration files and the model metadata, and Generation implements greedy decoding and top-k sampling.

The Models module wraps a converted Core ML package behind a LanguageModel type, and Core ML then decides whether to run it on the CPU, the GPU or the Neural Engine. Cuenca writes that flexible input shapes run on the CPU only, which is why Hugging Face’s exporters conversion tool switched to fixed sequence lengths, and that the Core ML path cached no attention keys and values at that point, both of which cap how long a context an app can work with.

Hugging Face states it tested the package with Llama 2, Falcon, StarCoder and the GPT family including GPT2, distilgpt, GPT-NeoX and GPT-J, and that encoder-decoder models such as T5 and Flan were not supported. The release came alongside swift-chat, a Mac app that loads a Core ML model and doubles as a model tester, plus converted weights for Llama 2 7B and Falcon 7B on the Hub. The package is licensed under Apache 2.0.

Microsoft Research published phi-1 on June 20, 2023, a Python code model with 1.3B parameters. The paper reports 50.6 percent pass@1 on HumanEval and 55.5 percent on MBPP, pass@1 being the share of problems the model solves with its first generated answer. The same table lists StarCoder at 15.5B parameters and 1 trillion training tokens with 33.6 percent and 52.7 percent, GPT-3.5 at 175B parameters with 47 percent, and CodeGen-Mono at 16.1B parameters and 577B tokens with 29.3 percent.

The training data comes to under 7B tokens in three parts. About 6B tokens are Python files pulled from The Stack and StackOverflow by a classifier the authors trained on roughly 100,000 GPT-4 ratings of how instructive a snippet is for someone learning to code. Under 1B tokens are synthetic Python textbooks written by GPT-3.5, and about 180M tokens are synthetic exercises with solutions, used only in the finetuning step that turns phi-1-base into phi-1.

Microsoft ran the training on 8 Nvidia A100 GPUs, reaching the base model in under 4 days and spending another 7 hours on the finetuning. That amounts to about 8 passes over the pretraining data and a little over 50B tokens seen, which the paper’s chart puts at 1090 GPU hours for the 1.3B run.

The same pipeline at 350M parameters yields phi-1-small, which the authors put at 45 percent on HumanEval and use to argue that parameter count still drives which capabilities appear at all. Microsoft posted phi-1 on Hugging Face in September 2023, where the model card lists the MIT license.

Artem Savkin published the first release of LLMFarm on June 14, 2023, an iOS and macOS app that loads a quantised model file kept on the device and holds a chat with it offline. The app is built on ggml and llama.cpp, and the README asks for iOS 16 or later and macOS 13 or later.

The app reads two llama.cpp file formats, the older ggjtv3 files with a .bin extension and GGUF files, according to the project FAQ. The first release covered LLaMA, GPT-NeoX and GPT-2 style models, with 3B builds of OpenLLaMa, StableLM and Dolly v2 among those it listed as working on an iPhone. The README has since added Gemma, Phi, Qwen, Mixtral, RWKV and LLaVA-style vision models, and marks most of them as running through Metal, Apple’s GPU interface, which it says does not work on Intel Macs.

Quantisation happens before a model reaches the phone, and the project’s on-device notes list 7B Llama 2 and Orca builds at 3-bit and 4-bit as the ones tried on an iPhone 12 Pro Max, with no published token rates. Each chat writes a JSON file of inference options, which the first release documented with defaults of temperature 0.8, top-k 40, top-p 0.95, a 2048-token context and a repeat penalty of 1.1. The app also offers tail free sampling, locally typical sampling, Mirostat, greedy decoding and grammar-constrained output. Models arrive either through an add model button that imports a file from local storage or through a download list inside the app, and the sandboxed app keeps them in its own models directory.

Memory sets the ceiling on an iPhone. The first release’s model table marked 3B models as working on an iPhone 12 Pro Max and 7B and 13B models as returning a “bad alloc” error, and the README said extended memory might be needed to solve it. The FAQ states that the Extended Virtual Addressing and increased-memory-limit entitlements are required to run models larger than 3B, and that the most common reason for a crash is lack of memory. LLMFarm is published under the MIT license, and the README now says the app is temporarily unavailable in TestFlight and the App Store.

Bo Peng and 29 co-authors published RWKV on May 22, 2023, an architecture that trains in parallel like a transformer but runs as a recurrent network when it generates text. They released pretrained weights in six sizes from 169M to 14B parameters, all trained on the Pile, and describe the 14B model as by far the largest dense RNN trained to that point.

A transformer keeps a cache of every token it has already processed, so both the memory it holds and the work it does for each new token grow as the context gets longer. RWKV carries a fixed-size state instead, which holds the same amount of data at token 10 and at token 10,000. The complexity table in the paper puts RWKV at O(Td) time and O(d) space for a sequence of T tokens, against O(T²d) time and O(T² + Td) space for a standard transformer, and the authors note that the arithmetic per token does not depend on context length at all.

The authors measured text generation on an NVIDIA A100 with 80 GB in float32 and report cumulative time that rises in a straight line for RWKV while the transformer baselines curve upward. In their chart, RWKV-4 3B reaches about 10 seconds of cumulative GPU time after 1024 tokens, where OPT 2.7B, GPT-Neo 2.7B, BLOOM 3B and Pythia 2.8B need roughly 57 to 63 seconds.

On zero-shot benchmarks the paper puts RWKV close to transformers of comparable size trained on the same data. RWKV-4 at 169M scores 65.07 on PIQA where Pythia 160M scores 62.68 and GPT-Neo 125M scores 63.06, and at 14.2B it leads an interpolated GPT-level baseline on five of seven common sense tasks while trailing it on WinoGrande and ARC-Easy. The authors list the trade-off as a limitation, since funnelling everything through one fixed state limits how well the model recalls small details from very long contexts, and they report that RWKV is more sensitive to prompt wording than a transformer.

Peng publishes the code as RWKV-LM under Apache 2.0, and the weights on Hugging Face carry the same license. The repository lists rwkv.cpp among community projects, a ggml-based implementation that runs 4-bit, 8-bit and float inference on CPU, while optimised iOS and Android inference sits in the README as a to-do rather than finished work. Peng states the goal in that list.

RWKV is a RNN and very friendly for edge devices. Let’s make it possible to run a LLM on your phone.

Qualcomm published a whitepaper, The future of AI is hybrid, in May 2023 arguing that generative AI inference has to be distributed between the cloud and edge devices rather than run in data centres alone. The company’s case rests on cost. It cites a Reuters report estimating that a generative AI search costs 10 times more per query than traditional search, and a Morgan Stanley note from February 2023 saying that with more than 10 billion search queries a day, even a small share moving to language models could add multiple billions of dollars a year.

On what can run locally, Qualcomm writes that models of 1 to 10 billion parameters, assuming INT4 weights, cover capabilities including text-to-image, dialog, image understanding and video understanding. It states that models with more than 1 billion parameters, such as Stable Diffusion, already run on phones with performance and accuracy similar to the cloud equivalent, and that models of 10 billion parameters or more are slated to run on devices in the near future. The chart in the paper puts the phone and laptop reach at about 10 billion parameters for 2023 and extends it further for 2024, while the cloud range spans the whole axis to 1000 billion.

The paper sets out three ways of splitting a workload. In the device-centric option the device runs most of the inference and an on-device neural network or rules-based arbiter decides when a query needs the cloud, with Qualcomm expecting laptops to run models of up to tens of billions of parameters. In the device-sensing option, speech recognition such as Whisper, computer vision and text-to-speech run on the phone while a cloud model does the language work, which the company argues cuts both compute and bandwidth.

The third option has device and cloud work on the same response through speculative decoding, a technique published by Google researchers in 2022. Qualcomm’s example has a draft model 7 to 10 times smaller than the target model generate four tokens on the device, which the cloud then checks by running the target model once across the four candidates, since a model like GPT-3 has to read all 175 billion parameters from memory to produce a single token. The company says its own early experiments and published results show 2 to 3 of the 4 draft tokens being accepted on average.

The only on-device demonstration the paper cites is Qualcomm’s own Stable Diffusion run on an Android phone from February 2023. Its other device claims are projections, including text-to-image models generating 3D textures on smartphones within a year and a 13 billion parameter LLaMA model reaching edge devices to voice avatars in XR. On energy the document argues only that edge devices run generative AI at a fraction of the cloud’s consumption once data transport is counted, without giving a figure. The whitepaper is Part I of two, announced in a Qualcomm blog post by Ziad Asghar and Jilei Hou, with Part II covering the company’s own products.

The MLC project released MLC LLM on May 1, 2023, a compiler and runtime that deploys language models natively on phones, browsers and consumer hardware rather than serving them from a data centre. The team pointed users to a TestFlight build to try the chat app on iOS.

MLC LLM compiles models through the Apache TVM Unity stack and maps them to the Vulkan and Metal graphics APIs, which the team says covers the majority of consumer platforms. Targets named at launch include Windows, Linux, macOS and iOS, plus AMD, NVIDIA, Intel and Apple GPUs and a Steam Deck. A companion project, WebLLM, offloads the same models into web browsers through WebGPU.

Three weeks later the team added RedPajama-3B across the same set of targets, with a browser demo and a command line tool for desktop machines.

The code is on GitHub under the Apache 2.0 license. The project now exposes an OpenAI-compatible API through a REST server as well as Python, JavaScript, iOS and Android bindings, and covers Android GPUs via OpenCL alongside CUDA and ROCm on the desktop.

Researchers at MBZUAI, the University of British Columbia and Monash University published LaMini-LM on April 27, 2023, a family of 17 distilled instruction-following models ranging from 61M to 7B parameters. The paper gives resource use as the motivation, stating that earlier distilled models of 7B to 13B parameters present “challenges for deployment in resource-constrained settings” and that the authors set out to make deployment in such settings easier. The paper names no target device and reports no memory footprint, no quantisation and no generation speed, and the authors state they trained and evaluated on V100 and A100 GPUs.

The models are fine-tuned on the LaMini instruction dataset, 2.58M instruction and response pairs that the team generated with gpt-3.5-turbo from prompts taken from self-instruct, P3, FLAN and Alpaca. The authors describe it as the largest instruction dataset of its kind at the time and 50 times the size of the dataset behind Alpaca. The students are 6 encoder-decoder models built on T5 and Flan-T5 and 11 decoder-only models built on GPT-2, GPT-Neo, Cerebras-GPT, GPT-J and LLaMA.

The authors evaluated the family on 15 NLP benchmarks with EleutherAI’s lm-evaluation-harness and report that LaMini-Flan-T5-248M performs on par with LLaMA-7B on the benchmark average, against a baseline with about 28 times as many parameters. They warn in the repository that their LLaMA figures are not comparable to the ones the LLaMA authors published, because they re-ran the evaluation themselves. A human rating of 114 user-oriented instructions follows the same size trend and leaves a gap to gpt-3.5-turbo, and the paper reports that the models do poorly on coding, maths and tasks that need reasoning.

MBZUAI publishes the weights and the instruction dataset on Hugging Face. The repository puts the code under Apache 2.0 and the models and the data under CC BY-NC 4.0, and states that both are intended for research use only. The paper was later accepted at the EACL 2024 main conference.

Bip-Rep published Sherpa on March 27, 2023, a Flutter app that runs llama.cpp on the phone itself, so the chat works with no network connection. The README asks users to bring their own weights, Meta’s 7B LLaMA converted to a ggml model file, which the app loads through a file picker, and version 2.1.0 added Vicuna support and macOS builds next to the Android and Windows ones. The project publishes no tokens per second figure and states no minimum memory, and the only device it names is a OnePlus 7 with 8 GB of RAM in the demo video below, which the README says was recorded at real speed. The code is MIT licensed.

Development stopped three weeks after it started. All 43 commits fall between March 27 and April 14, 2023, the last of the seven releases went out on April 14, 2023, and the llama.cpp fork the app builds against was last touched on April 12, 2023, four months before llama.cpp moved to the GGUF format. Sherpa reads the older ggml files rather than the GGUF models that have been standard since, and requests for GGUF support have gone unanswered, with 23 of the 29 issues ever filed still open. A contributor answered one of them by writing that the original developers had abandoned the project, and pointed to his own fork, Maid.