https://llmobile.kavents.com/tags/llama/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/mlperf-inference-mobile/https://llmobile.kavents.com/ticker/mlperf-inference-mobile/Mon, 15 Jun 2026 17:00:00 +0200MLCommons announced MLPerf Mobile v6.0 on June 15, 2026, adding generative AI tests that run Llama on an Android device and report token throughput. The new workload comes in 1B, 3B and 8B parameter sizes, using Llama 3.2 1B Instruct, Llama 3.2 3B Instruct and Llama 3.1 8B Instruct. Each size runs twice, once against TinyMMLU, a small multiple-choice knowledge set, and once against IFEval, which checks whether a model obeys the instructions in a prompt, according to the task configuration in the app repository. MLCommons says the tests run on devices with enough memory through the CPU without tailored acceleration, and that v6.0 adds NPU-accelerated execution of Llama 3.1 8B Instruct on Qualcomm Snapdragon 8 Elite Gen 5. The v6.0 app release of June 9, 2026 also brings support for MediaTek Dimensity 9500 chips and updates support for the Samsung Exynos 2600. The rest of the suite covers image classification with MobileNetV4 on ImageNet, object detection on COCO 2017, segmentation with MOSAIC on ADE20K, language understanding with MobileBERT on SQuAD 1.1, super resolution with EDSR, and Stable Diffusion 1.5 on COCO captions, as the benchmark page lists them. Almost every test runs in the single stream scenario, where the score is the 90th percentile latency for one query at a time, and image classification also runs an offline pass scored as throughput. Stable Diffusion, the first generative test in the suite, arrived in the v4.1 app release of February 11, 2025. Mostafa El-Khamy, co-chair of the MLPerf Mobile working group, had set out the plan in May 2024. Looking ahead, the MLPerf Mobile group is working to add generative AI tasks to a future version of the benchmark suite. Results come from chip and device makers that run the app and submit their logs. The published v6.0 results hold one submission in the closed division, from Samsung on an Exynos 2600 phone, and it covers the vision tests, language understanding, super resolution and Stable Diffusion rather than the new Llama ones. Samsung’s logs give a 90th percentile single stream latency of 0.56 ms for image classification, 0.83 ms for MobileBERT and 1.87 s for one Stable Diffusion image. The v5.0 round carried submissions from Samsung and Qualcomm Innovation Center. Screenshots: Samsung's submission to MLPerf Mobile v6.0, published by MLCommons. Submissions are held back until review closes under the MLCommons submission rules, which keep every result confidential inside a review committee after the deadline and let submitters file objections that the committee then decides. MLCommons publishes the accepted runs in the results table on the MLPerf Inference Mobile page, with the logs and calibration notes in a per-round repository on GitHub. The app is available through Google Play, the Apple App Store and the MLPerf Mobile repository under the Apache 2.0 license. Source: https://mlcommons.org/2026/06/mlperf-mobile-v6/ Read the article: https://llmobile.kavents.com/ticker/mlperf-inference-mobile/BenchmarksAndroidLlamaNPUQualcommhttps://llmobile.kavents.com/ticker/quant-npu/https://llmobile.kavents.com/ticker/quant-npu/Tue, 19 May 2026 12:48:00 +0200Eight authors published Quant.npu on May 19, 2026, a way of quantising 1B to 3B language models so they run on a phone’s neural processing unit with no rescaling at runtime. On a Qualcomm SM8650 NPU they measure their 4-bit weight, 8-bit activation build of Llama-3.2-3B-Instruct finishing up to 15.1 percent faster than the same model under ExecuTorch’s 4-bit per-block weights with 16-bit activations, for about 1 percentage point of zero-shot accuracy. Across every model and task in the paper the authors put the average cost at a 2.58 percent accuracy drop and a 1.23 rise in perplexity. Quantisation stores a tensor as small integers plus a scale factor that maps them back to real values. Most published 4-bit methods, including the QuaRot and SpinQuant work Quant.npu is measured against, work that scale out while the model runs by scanning each activation tensor for its largest value. A mobile NPU compiles its graph before the model ships and expects every scale to be fixed by then, because that scan is a reduction the hardware is poor at, at least 7 vector instruction cycles for a 128-element 8-bit vector by the authors’ count, and because two of the four rotation steps these methods use against outliers need floating-point matrix multiplications during inference, which the paper puts at roughly double the compute. Anything left dynamic therefore has to run outside the compiled integer pipeline, and the authors report that simply freezing a model optimised under dynamic quantisation costs 15.61 percent accuracy on SmolLM2-1.7B-Instruct. Quant.npu learns the scales during optimisation instead and folds its two rotation matrices into the weights offline, so the shipped graph is integer arithmetic end to end. In the paper’s main table for Llama-3.2-3B-Instruct at 4-bit weights and 8-bit activations, that lifts average zero-shot accuracy across six benchmarks to 58.27 percent against 46.23 for SpinQuant and 47.66 for QuaRot in the same static setting, with perplexity on C4 falling from 28.78 to 19.16. Against the slower ExecuTorch configuration it beats on latency, though, Quant.npu is the one behind, scoring lower on five of the six benchmarks, 1.08 points lower on average and 0.97 higher in perplexity, and FP32 stays ahead of both at 61.3 percent. On the newer SM8750 NPU the authors report peak memory of 1876 MB to 1985 MB for their 3B builds against 2216 MB to 2435 MB for ExecuTorch, and prefill of 969.72 tok/s for Qwen2.5-3B-Instruct on the DroidTask set against 808.10. They name two limits of their own, that the pipeline still keeps a few operations such as the SiLU activation at 16 bits to hold accuracy, which they call a bottleneck against a fully low-bit pipeline, and that they use no method for picking calibration data even though the optimisation is sensitive to it. Quantising Qwen3-1.7B takes about 2.5 hours on NVIDIA A40 GPUs, and the preprint announces no code release. Source: https://arxiv.org/abs/2605.20295 Read the article: https://llmobile.kavents.com/ticker/quant-npu/QualcommNPUQuantisationLlamaResearchhttps://llmobile.kavents.com/ticker/flexserve/https://llmobile.kavents.com/ticker/flexserve/Tue, 10 Mar 2026 01:31:25 +0100Researchers at Shanghai Jiao Tong University published FlexServe on March 10, 2026, a serving system that keeps an on-device language model’s weights and the text going through it inside ARM TrustZone’s secure world. Running Llama 3.2 3B, Llama 3.1 8B, Qwen3 1.7B and Qwen3 8B quantised to 8-bit on a NanoPC-T6 board with a Rockchip RK3588, 16 GB of memory and a 6 TOPS NPU, the authors measure time to first token 4.41 percent above the same inference run without any protection, averaged over their prompt lengths. Against a TrustZone baseline that already uses a secure NPU and a loading pipeline, they report FlexServe 1.43 to 2.42 times faster, 1.85 times on average. TrustZone splits the chip into a normal world, where Android and every app runs, and a secure world the normal world cannot read. FlexServe puts the inference engine, the weights, the prompt, the generated text and the KV cache on the secure side, so a kernel an attacker has taken over cannot read any of it out of memory, and the authors justify that assumption with Linux’s 40 million lines of code and 9756 CVEs. Both halves are worth stealing in their account, since a vendor’s fine-tuned weights cost real money to train yet sit on a stranger’s device, and the apps calling a shared model feed it chat history and screen contents. The authors name the boundary they cannot cross, which is that an app in the normal world still receives its own input and output there, so a compromised kernel can take those, and only the weights stay protected in that case. Plain TrustZone is slow at this because it hands out secure memory as a few large contiguous regions. Reserving 8 GB at runtime therefore means the kernel’s contiguous allocator compacting scattered pages, which the authors time at 6440.67 ms against 568.58 ms for FlexServe’s page-granular secure memory, and the secure world cannot normally touch the NPU at all, leaving 30.06 s of CPU computation where the NPU needs 1.94 s. FlexServe carves secure pages out with the stage-2 page tables that ARM’s virtualisation extension already provides, and flips the NPU between protected and unprotected mode in 0.21 ms. Handing 8 GB back drops from 732.85 ms to 80.50 ms, because the pages are released lazily rather than zeroed in one pass. A scheduler decides which models stay resident when several share the device, holding the earliest layers of each in secure memory so the pipeline does not stall and prefetching the next model while the current one is still generating. With a 4 GB cache and requests drawn at random from UltraChat, OpenAssistant, Dolly and Alpaca, the authors measure pairs of models answering 8.84 times faster than the plain secure-world baseline and 2.53 times faster than the optimised one. Across ten agent workflows, where one model’s finished answer feeds the next, the gains reach 24.30 times and 4.05 times, averaging 14.15 and 2.94. Serving one phone-side model to several callers is also what ElastiLM works on, by resizing the shared model to each request’s latency budget rather than isolating it. The protection costs the rest of the device something as well. A SQLite instance holding 3 GB in the normal world runs 1.31 times slower while FlexServe holds 8 GB in the background, against 3.27 times slower with the contiguous allocator, and the authors put the standing virtualisation cost at 2.46 percent on SPEC CPU, which they switch off when no secure inference is running. They list the prototype’s limits themselves, a development board rather than a phone, Llama 3 and Qwen3 series models only, no GPU in the pipeline, and an NPU runtime slow enough that computation becomes the bottleneck, so a faster NPU would widen their lead. Side-channel attacks, physical attacks and denial of service are out of scope, and the paper names no code release. Source: https://arxiv.org/abs/2603.09046 Read the article: https://llmobile.kavents.com/ticker/flexserve/ResearchArmNPUMemoryLlamahttps://llmobile.kavents.com/ticker/confidant-collaborative-training/https://llmobile.kavents.com/ticker/confidant-collaborative-training/Mon, 03 Nov 2025 10:00:00 +0100Researchers at Zhejiang University, Zhejiang University of Technology and Hangzhou Dianzi University published Confidant in the ACM MobiCom 2025 proceedings on November 3, 2025, a day before the conference opened in Hong Kong. The framework fine-tunes a language model across several off-the-shelf devices instead of one, and the authors fine-tuned Phi2-2.7B on the Alpaca instruction dataset in 40.1 hours using three of them. Their write-up of that run names a Redmi K50 phone, a Core i7-13650HX laptop and a MacBook Pro with an M1 Pro, while the paper’s abstract and conclusion describe the same result as three mobile devices. The same trio fine-tuned BERT-Base in 2.2 hours. Confidant cuts a model into consecutive slices, gives each device one slice, and passes activations from one device to the next, so no device ever holds the whole model. Batches flow through that chain as a pipeline, with each device starting a new batch while the devices behind it are still working on earlier ones. The device that holds the training data acts as coordinator and recalculates where the cuts fall during training, using each participant’s measured compute time, free memory and bandwidth, and it discards any split whose slice would not fit in a device’s memory. Inside a device, a second scheduler profiles the CPU and the GPU on varying numbers of attention heads and gives each processor as many heads as it can finish in roughly the same time, which is what the authors credit for running up to 1.94 times faster than a capacity-aware pipeline baseline they implemented themselves, and up to 3.82 times faster than the best single device, both on GPT-2 Medium. Diagram: Chen et al., Figure 3 of the paper, licensed CC BY 4.0. Splitting the model is what brings the footprint per device down. The paper’s memory table puts fine-tuning BERT-Base at a batch size of 8 at 5.5 GB on a single Redmi K50, against a per-device average of 3.17 GB across three devices and 2.43 GB across four. Larger models leave no single-device figure to compare with, because a Redmi K50 fits only 1 of Phi2-2.7B’s 32 transformer blocks at a batch size of 4, and the per-device averages of 15.31 GB for Phi2-2.7B and 32.07 GB for LLaMA3-8B across three devices are carried by the two laptops rather than the phone. A separate table puts the memory PyTorch needs to fine-tune Phi2-2.7B at a batch size of 8 at 46.5 GB, next to the 8 GB to 12 GB of the four phones it lists. The paper’s prose says Confidant almost does not cause any accuracy drop, while its own model performance table puts the three-device runs at a test loss of 0.66 on Phi2-2.7B and 3.55 on LLaMA3-8B, against 0.52 and 2.86 for fine-tuning the same models the conventional way on a single machine, where a lower loss is better. The four-device runs land at 0.76 and 2.85 on that same comparison, and BERT-Base token classification accuracy holds at 97.66% against 97.70%. Keeping data on the device is the stated motivation. The authors write that training with Confidant does not require sharing raw data and that devices exchange only intermediate outputs and model gradients, which they present as the privacy argument for their two scenarios, a household assistant fine-tuned on family data during idle hours and a conference assistant fine-tuned on attendees’ phones. They also state that fine-tuning on a single cloud server would be faster and more straightforward, and that their aim is to use idle mobile hardware instead. The Android side is built on Alibaba’s MNN framework with laptops running PyTorch, about 10,900 lines of code in total, the paper is published under a Creative Commons Attribution 4.0 license, and the authors wrote that they planned to release the code in late 2025. Source: https://dl.acm.org/doi/10.1145/3680207.3723487 Read the article: https://llmobile.kavents.com/ticker/confidant-collaborative-training/ResearchMemoryPhiLlamaAndroidhttps://llmobile.kavents.com/ticker/roma-qlora-accelerator/https://llmobile.kavents.com/ticker/roma-qlora-accelerator/Mon, 17 Mar 2025 10:44:00 +0100Researchers at Shanghai Jiao Tong University, Peking University and Microsoft Research Asia described ROMA on March 17, 2025, an accelerator design that holds a quantised language model in on-chip read-only memory. The paper reports results from Verilog synthesis against TSMC’s 7 nm standard cell library, not from a fabricated chip and not from an FPGA prototype, so every speed, area and power figure in it is a toolchain estimate rather than a measurement on hardware. On that basis the authors report peak decoding of 31,800 tok/s for a 4-bit 3B Llama model with an empty KV cache. The design splits storage by how often the weights change. The frozen base model, quantised to 2-bit or 4-bit, sits in ROM, which cannot be rewritten once the chip has been manufactured, while the small trainable LoRA adapter and the KV cache sit in SRAM, which can. The authors argue that a base model is stable enough for that, with updates expected over years rather than months, and that adaptation to new tasks happens in the adapter instead. What ROM buys is density, and the paper puts a ROM bit cell at roughly a third of the area of an SRAM bit cell in TSMC’s 7 nm memory compiler, enough to fit the whole quantised model on chip without external memory. The synthesised configuration carries 1.86 GB of ROM and 304 MB of SRAM at 500 MHz, with an area of 503.7 mm² and power of 33.1 W, the area and latency coming from Synopsys Design Compiler and the power from Synopsys PrimeTime PX. It holds either a 4-bit 3B or a 2-bit 8B Llama model entirely on chip, and for the 3B model with a rank-16 adapter the authors report a time to first token of 5.6 ms at 256 input tokens and 140.2 ms at 4K, with decoding staying above 10,000 tok/s once 4K tokens are cached. They also compare against measured hardware, an Intel i5-1135G7 running llama.cpp at 6.8 tok/s and an Nvidia RTX 4090 running TensorRT-LLM at 219 tok/s on the same 4-bit 3B model, against 20,078 tok/s for the ROMA estimate at a 256-token input. Most of the paper’s engineering goes into shrinking the ROM. A structure the authors call B-ROM stores the constant bits in the chip’s wiring instead of in transistors, which cuts the transistor count to about a quarter and the ROM area by around 40 percent against a standard ROM, and a fused cell layout places that wiring-heavy structure in the same physical area as the transistor-heavy compute logic. The paper states no tapeout, no fabrication schedule and no cost figures, and it names the fixed base model as the price of the approach, since ROM content is set at manufacture. It is posted under the arXiv perpetual non-exclusive license. Source: https://arxiv.org/abs/2503.12988 Read the article: https://llmobile.kavents.com/ticker/roma-qlora-accelerator/ChipsMemoryQuantisationResearchLlamahttps://llmobile.kavents.com/ticker/flower-intelligence/https://llmobile.kavents.com/ticker/flower-intelligence/Wed, 12 Mar 2025 10:00:00 +0100Flower Labs released Flower Intelligence on March 12, 2025 as a preview, an open source inference library that runs a language model on the user’s own device and only moves the work to a remote service if the app and the user allow it. The company shipped SDKs for Swift and TypeScript, and the TypeScript build defaults to Llama 3.2 1B in 16-bit precision. The library picks a local engine from the environment it finds itself in. The TypeScript package depends on MLC’s WebLLM, which executes models in the browser through WebGPU, and on Hugging Face transformers.js, which it uses when the code runs under Node.js rather than in a page. The Swift package is built on MLX Swift and Hugging Face swift-transformers, and declares iOS 16 and macOS 14 as its minimum platforms, so local inference on Apple hardware goes through MLX. Flower Labs’ model reference covers Llama 3.2 at 1B and 3B, Llama 3.1 8B, SmolLM2 at 135M, 360M and 1.7B, and DeepSeek R1 Distill Llama 8B, most of them as 4-bit builds. Which of those run on device depends on the backend. Node.js is limited to the 1B and SmolLM2 entries, while the 8B models run only in the browser and on the MLX Swift path, and Llama 3.2 3B in 16-bit is MLX Swift only. Remote handoff is off by default. An application has to set the remoteHandoff property and supply an API key before the library will contact Flower Confidential Remote Compute, and even with both in place a chat call tries a local engine first and retries remotely only when no local engine matches the model or the local run fails. Individual requests can be pinned with forceLocal or forceRemote. Flower Labs describes the remote service as a private extension of the device that costs the developer no extra work, and named Mozilla Thunderbird as an early adopter building its Thunderbird Assist feature on the library. Ryan Sipes, Managing Director for Mozilla Thunderbird, tied that choice to the mail client’s user base in the announcement. Our 20 million users expect data privacy from every feature we build. Both SDKs are published under the Apache 2.0 license, the TypeScript one as @flwr/flwr on npm and the Swift one as a package in the Flower repository. Flower Labs called the release inference-only and listed fine-tuning, RAG and pre-training as later steps, with the remote compute service available on application for early access. Source: https://flower.ai/blog/2025-03-12-flower-intelligence Read the article: https://llmobile.kavents.com/ticker/flower-intelligence/Open sourceDeveloper toolsLlamaWebGPUApple Siliconhttps://llmobile.kavents.com/ticker/exolabs-benchmarks/https://llmobile.kavents.com/ticker/exolabs-benchmarks/Wed, 25 Dec 2024 19:55:00 +0100EXO Labs opened a public benchmark site on December 25, 2024, as the first entry in a twelve-part series, publishing generation speed and time to first token measured on real consumer devices. EXO states these are measurements rather than estimates. In the data behind the site, an iPhone 15 Pro with 8 GB of RAM generates 14.0 tok/s on Llama 3.1 8B and takes 1.67 seconds to return its first token. A Galaxy S24 Ultra with 12 GB manages 5.6 tok/s on the same 8B model, with 4.16 seconds to the first token, according to EXO’s published results. On the smaller Llama 3.2 1B the iPhone reaches 56.1 tok/s and the Galaxy 22.5 tok/s. A single Mac mini M4 Pro with 24 GB sits at 49.3 tok/s on Llama 3.1 8B, which puts the phone at under a third of the Mac’s rate on the same model. EXO splits a model into contiguous slices of its layers and hands each slice to a different device, so a model too large for one machine can run across several. The activations passed from one device to the next are small, under 4 KB for Llama 3.2 3B, and EXO says the bottleneck is network latency rather than bandwidth. Pooling phones does not speed up a single reply. A cluster of two iPhone 15 Pros and one Galaxy S24 Ultra runs Llama 3.1 8B at 7.0 tok/s, below the 14.0 tok/s of one iPhone alone, which EXO attributes to network overhead when the model already fits on a single device. Three Mac mini M4 Pros run Llama 3.3 70B at 4.1 tok/s, a model that does not fit in the 24 GB of any one of them. Every entry carries a timestamp, a git commit and the identifier of the automated run that produced it, and the site is free to browse. EXO says it is expanding the suite towards mixed device types and different quantisations, and the exo runtime itself is on GitHub under Apache 2.0. Source: https://blog.exolabs.net/day-1/ Read the article: https://llmobile.kavents.com/ticker/exolabs-benchmarks/BenchmarksiPhoneAndroidLlamaApple Siliconhttps://llmobile.kavents.com/ticker/amd-llama-135m/https://llmobile.kavents.com/ticker/amd-llama-135m/Fri, 27 Sep 2024 19:00:00 +0200AMD released AMD-Llama-135M on September 27, 2024, its first small language model, at 135M parameters and trained from scratch on AMD’s own accelerators. AMD states that pretraining used 670B tokens drawn from SlimPajama and Project Gutenberg and took six days on four Instinct MI250 nodes holding four MI250 accelerators each. A second variant, AMD-Llama-135M-code, adds 20B tokens from the Python subset of StarCoder over another four days on the same hardware. That code variant exists to work as a draft model for speculative decoding, where a small model guesses several tokens in one pass and a larger model then checks them, so the larger model can accept several tokens per forward pass rather than producing one at a time. AMD paired it with Meta’s CodeLlama-7b and measured throughput on the HumanEval coding benchmark. AMD reports about 2.8x higher throughput on an Instinct MI250 accelerator, about 3.88x on the CPU of a Ryzen AI processor, and about 2.98x with the 7B model running 4-bit on the Ryzen AI NPU while the draft model ran on the CPU, each against the same setup without speculative decoding. AMD’s footnotes name a Ryzen 9 PRO 7940HS as the test machine for the Ryzen AI numbers. Diagram: AMD, showing its own measured speedups. AMD published the weights on Hugging Face and the training, dataset and inference code on GitHub, under the Apache 2.0 license. AMD states the training code builds on TinyLlama and uses PyTorch FSDP for distributed training across nodes. Source: https://www.amd.com/en/developer/resources/technical-articles/introducing-amd-first-slm-135m-model-fuels-ai-advancements.html Read the article: https://llmobile.kavents.com/ticker/amd-llama-135m/AMDLlamaNPUOpen weightsOpen sourcehttps://llmobile.kavents.com/ticker/llama-stack/https://llmobile.kavents.com/ticker/llama-stack/Wed, 25 Sep 2024 19:30:00 +0200Meta announced Llama Stack at Connect on September 25, 2024, a set of REST APIs plus client SDKs for Python, Node, Swift and Kotlin that cover the same Llama features wherever the model runs. The repository defines APIs for Inference, Safety, Memory, Agentic System, Evaluation, Post Training, Synthetic Data Generation and Reward Scoring, and lists PyTorch ExecuTorch as the on-device provider, covering inference and agents on iOS. The models announced at the same event are covered separately. Llama Stack separates providers from distributions. A provider is the implementation behind an API, which can be local code or a pointer to a remote service, and a distribution assembles providers behind one endpoint. Meta’s README states that providers can be mixed, that a hobbyist can serve a small model locally and use a cloud provider for a large one, and that the higher level APIs the app works with do not change, including across the server and mobile-device boundary. Diagram: Meta. From the Llama Stack repository documentation. On iOS the local provider is a Swift class called LocalInference, which the documentation describes as powered by ExecuTorch. It implements the same Inference interface the Swift client uses for remote servers, so an app that already streams chat completions or agent turns keeps its calling code and swaps the service it was given. Developers export the model to an ExecuTorch .pte file, bundle it with tokenizer.model in the app, and link the XNNPACK, Core ML and MPS backends plus the quantized and optimised kernels, with Meta’s own example loading a SpinQuant build of Llama 3.2 1B. The Swift package is added through Xcode’s package manager and generates its types from the Llama Stack OpenAPI spec, and Meta listed its first release on October 10, 2024. The same iOS documentation stated that on-device inference for Android was coming soon, and pointed developers to ExecuTorch’s own Android demo app in the meantime. Llama Stack, the Swift client and the Kotlin client are published under the MIT license. Update, December 6, 2024. Meta released version 0.0.54 of the Kotlin client, which adds on-device inference on Android. The README names ExecuTorch as the local inference distributor, has developers run a script that fetches a prebuilt executorch.aar with the XNNPACK delegate into the app’s libs directory, and shows the only difference between the two modes as the client builder, LlamaStackClientLocalClient with a model and tokenizer path against LlamaStackClientOkHttpClient with a base URL. The chatCompletion call that follows is identical. Source: https://github.com/meta-llama/llama-stack Read the article: https://llmobile.kavents.com/ticker/llama-stack/MetaLlamaiOSAndroidDeveloper toolshttps://llmobile.kavents.com/ticker/llama-3-2-1b-3b/https://llmobile.kavents.com/ticker/llama-3-2-1b-3b/Wed, 25 Sep 2024 19:00:00 +0200Meta released Llama 3.2 on September 25, 2024, including text-only 1B and 3B models built for phones and edge hardware. Both carry a 128K token context window and are aimed at summarisation, instruction following and rewriting that run locally, with the data staying on the device. Meta built them by structured pruning from Llama 3.1 8B, then recovered quality through knowledge distillation using logits from the 8B and 70B models during pretraining. The company reports the 3B model ahead of Gemma 2 2.6B and Phi 3.5-mini on instruction following, summarisation, prompt rewriting and tool use, and puts the 1B model on a par with Gemma. In Meta’s own table the 3B model scores 77.4 on IFEval against 61.9 for Gemma 2 2B and 59.2 for Phi-3.5-mini, and 67.0 on BFCL V2 for tool use against 27.4 and 58.4. Table: Meta. The company measured the Gemma and Phi results itself. The models shipped with day-one support for Qualcomm and MediaTek silicon and run on Arm, which Meta says covers 99 percent of mobile devices. Weights are on llama.com and Hugging Face, with deployment paths through PyTorch ExecuTorch for devices and Ollama for single-node setups, and the company lists more than 25 partner platforms at launch. Source: https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ Read the article: https://llmobile.kavents.com/ticker/llama-3-2-1b-3b/MetaLlamaOpen weightsQualcommMediaTekhttps://llmobile.kavents.com/ticker/on-device-collaborative-lm/https://llmobile.kavents.com/ticker/on-device-collaborative-lm/Sat, 21 Sep 2024 00:34:00 +0200Four researchers at EPFL published CoMiGS on September 20, 2024, a scheme in which many devices fine-tune a language model together, with some of the added model parts shared across everyone and the rest kept on the device that trained them. In a simulation of four users, each holding Wikipedia articles in a different language, the authors measure test perplexity at 47.19 with GPT-2 124M as the base model, against 54.38 when each user trains alone on local data and 58.80 for federated averaging, which pools every user’s update into one shared model. Perplexity measures how surprised a model is by the next token, so lower is better. The paper appeared at ICML 2025. Nobody retrains the base model. Each device trains small LoRA adapters on top of pretrained weights that stay frozen, and CoMiGS sorts those adapters into two kinds. Generalist adapters are averaged across all devices at every communication round, specialist adapters and a small router never leave the device, and the router decides per token which adapters to use. The authors train that router against a separate validation set rather than the training data, and report that in the first layer of the network common function words such as “and”, “a” and “the” go mostly to the generalists, while for the users trained on maths and programming text the domain-specific terms in the last layer go mostly to the specialists. Nothing here ran on a phone. The appendix states that every experiment except the centralised baseline ran on a single NVIDIA A100-SXM4-40GB, with four simulated users holding between roughly 400,000 and 1 million training tokens each, drawn from multilingual Wikipedia, SlimPajama, AG News and Common Corpus, and averaged over three seeds. The base models are GPT-2 124M and Llama 3.2 1B. Results in the paper’s table are mixed rather than uniform, with CoMiGS ahead of both baselines on multilingual Wikipedia, SlimPajama and Common Corpus, and behind federated averaging on AG News, at 33.53 against 31.84 with GPT-2 and 16.31 against 15.86 with Llama 3.2 1B, which is the one setup where the test data does not follow the training distribution. The second set of experiments gives users different numbers of adapters, two to four per device with one of them always a generalist, which is how the method is meant to cope with devices of unequal capacity. Against HetLoRA and FlexLoRA, two methods that vary the LoRA rank per device instead, the authors report lower perplexity on every in-distribution task, for example 18.68 against 21.41 and 24.63 on Common Corpus with Llama 3.2 1B. They put the overhead of their extra router at 1.25 percent more computation per forward pass and 1.25 percent more memory, with communication per round halved against federated averaging because only the generalist weights are sent. The paper names its own limits. The authors write that differential privacy on the aggregated generalist weights is something they “do not pursue here”, that robustness against attackers is beyond the scope of the work and that their aggregation carries no guarantee against adversarial parties, and that there is a limit to how much the generalists can prevent overfitting when the local task is easy. They also note that fine-tuning Llama 3.2 1B on multilingual Wikipedia or SlimPajama produced negligible improvements, which they attribute to overlap with its pre-training data, and that is why the Llama runs use Common Corpus instead. The code is on GitHub under the Apache 2.0 licence. Diagram: Fan et al., Figure 2 of the paper, licensed CC BY 4.0. The router and the specialists stay on the device. Source: https://arxiv.org/abs/2409.13931 Read the article: https://llmobile.kavents.com/ticker/on-device-collaborative-lm/Mixture of expertsResearchOpen sourceLlamahttps://llmobile.kavents.com/ticker/elastic-on-device-llm-service/https://llmobile.kavents.com/ticker/elastic-on-device-llm-service/Sun, 08 Sep 2024 08:32:08 +0200Researchers at Peking University and Beijing University of Posts and Telecommunications published ElastiLM on September 8, 2024, a phone-side language model service that resizes itself for every request. Running LLaMA-7B on a Redmi K60 Champion Edition with a Snapdragon 8 Gen 2, the authors measure 0.31 seconds to switch from one sub-model to another, against 8.3 seconds for swapping in a separately pretrained model of the right size and 6.2 seconds for the pruning method LLMPruner. The setup the paper assumes is one model held by the operating system that every app calls, the way Android AICore serves apps like Gboard smart reply. Those callers want different things from it, and the authors put an always-on voice assistant that needs an answer almost immediately next to a background screen-event recorder that can wait. ElastiLM takes a latency budget along with each request and then picks how much of the model to run and how much of the prompt to keep, in steps of 10 percent, rather than running one fixed model at one fixed speed. Accuracy is what pays for the speed. Cutting the prompt drops words the model might have needed and pruning removes whole attention heads and MLP neurons, and the split matters as much as the total, since the authors show one science question that half the prompt with an 80 percent sub-model answers correctly while 80 percent of the prompt with a half-size sub-model gets wrong. Preparing a model costs 68.3 GPU hours on a server with NVIDIA A40s, which the authors price at about $100 of rented time and put at 21.7 to 68.2 hours more than the pruning baselines. The service runs on four big CPU cores, and the authors state that as of August 2024 no mobile GPU or NPU could run ElastiLM or its baselines end to end, so their accelerator figures are projected from profiled latency rather than measured on a device. Across the Redmi K60 Champion, a Mi 14 and a Redmi K70 Pro, the last two on Snapdragon 8 Gen 3, the authors report ElastiLM answering more requests correctly than seven baselines by 6.60 to 14.83 percent in absolute accuracy, 10.45 percent on average, on traces of 600 requests drawn from six datasets. Peak memory for LLaMA-7B sits at 15 to 17 GB, level with the baselines, while keeping a dedicated model per latency budget would take 29.3 GB or 76.3 GB and runs out of memory on all three phones, which carry 19 GB to 24 GB of RAM. Weight-only quantisation to 8 bits costs almost no accuracy and 4 bits about 3 percent absolute, by the authors’ measurement. The paper appeared at MobiCom 2025 in Hong Kong in November 2025 and is on arXiv under a CC BY 4.0 license. Charts: Yin et al. Online overhead measured on a Redmi K60 Champion Edition, with the dashed line marking 1 percent of average time to first token. Source: https://arxiv.org/abs/2409.09071 Read the article: https://llmobile.kavents.com/ticker/elastic-on-device-llm-service/ResearchAndroidQualcommLlamahttps://llmobile.kavents.com/ticker/torchchat/https://llmobile.kavents.com/ticker/torchchat/Tue, 30 Jul 2024 18:00:00 +0200PyTorch released torchchat on July 30, 2024, a small codebase showing how to run Llama 3 and other language models on laptops, desktops and phones. For the phone path the team reports more than 8 tok/s for Llama 3 8B Instruct on a Samsung Galaxy S23 and on iPhone, using 4-bit GPTQ quantisation through ExecuTorch. Diagram: PyTorch. The repository names iOS 17 or later with at least 8 GB of RAM as its requirement, which it narrows to iPhone 15 Pro and newer or an iPad with Apple silicon, and on Android any device covered by the XNNPACK CPU backend, with 12 GB of RAM and 20 GB of storage recommended. Its model table marks Llama 3.2 1B and 3B, Llama 3.1 8B, Llama 2 7B, Mistral 7B, IBM Granite 3.1 and DeepSeek R1 Distill 8B as mobile friendly. The 11B vision version of Llama 3.2 and the 13B and 70B Llama 2 checkpoints carry no such mark. Getting a model onto a phone is one export step that turns a downloaded checkpoint into a .pte file for ExecuTorch, the PyTorch edge runtime that reached version 1.0 in October 2025. The mobile preset shipped in the repository quantises the embedding table to 4-bit in groups of 32 values and the linear layers to 4-bit weights whose activations are quantised to 8-bit at runtime, in groups of 256. The quantisation guide offers 4-bit and 8-bit embeddings at group sizes from 32 to 256 across all execution modes, lists the dynamic-activation scheme as the supported one for ExecuTorch, and notes that smaller groups preserve accuracy while larger ones run faster. Both demo apps are sample harnesses rather than shipped products. On iOS the repository points at the ExecuTorch LLaMA example project in Xcode, which needs a provisioning profile carrying Apple’s increased memory limit entitlement, after which the model and tokenizer files are copied into the app’s folder on the device. On Android the repository carries its own app skeleton for Android Studio, built against a prebuilt ExecuTorch library, with the model and tokenizer pushed to the phone over adb. The other two paths in torchchat do not touch phones. One runs the model in Python through a chat and generate command line, a browser interface or a local server, the other compiles it ahead of time with PyTorch’s AOTInductor into a binary that a C++ runner loads on a desktop or server. The code is under a BSD 3-Clause license, and PyTorch halted active development on May 19, 2025, saying vLLM had become the dominant option for server-side inference. Source: https://github.com/pytorch/torchchat Read the article: https://llmobile.kavents.com/ticker/torchchat/PyTorchLlamaAndroidiOSQuantisationhttps://llmobile.kavents.com/ticker/mobile-foundation-model-as-firmware/https://llmobile.kavents.com/ticker/mobile-foundation-model-as-firmware/Wed, 29 May 2024 15:00:00 +0200Researchers at Beijing University of Posts and Telecommunications proposed that a phone ship one shared multimodal model instead of letting every app bundle its own, in a paper published in the ACM MobiCom 2024 proceedings on May 29, 2024. The operating system and the hardware co-manage that model like firmware, unchangeable by apps or by the OS itself, exposed to applications as a system service, and each app reaches it through a small adapter fine-tuned offline for its own task. Their prototype, called M4, holds 9.2B parameters and needs 7.5 GB of peak memory, and the authors report it reaching accuracy comparable to purpose-built models on 85% of the 50 datasets in a benchmark they assembled from 38 mobile AI tasks across five input types. What the shared model replaces is one small model per app per task. The paper’s baselines are 50 task-specific models of 1M to 500M parameters each, one per dataset, against which M4’s adapters run from 1,000 to 10 million parameters, so each added task costs under 10 MB. Measured on an Nvidia Jetson Orin NX, 4-bit M4 needs 6.1 GB of storage to serve all 50 tasks against 15.2 GB for the 50 separate models, with the crossover at about 15 tasks, and 7.5 GB of peak memory against roughly five times that. The authors state that on a device with 12 GB of memory the 4-bit model plus all 50 adapters fits, where only 20 of the 50 task-specific models would. The prototype is slower than the models it replaces. On the Jetson Orin NX with 16 GB, the authors measured M4 averaging 18 times the inference latency of the task-specific models across the 50 tasks and 19 times the energy, 3.6 s against 0.2 s. On a Pixel 7 Pro CPU they measured an average of 6.8 s against 0.54 s, and their per-task breakdown puts image classification at 2.10 s and question answering at 6.34 s to the first token and 0.24 s per token after it. They state that M4 cannot currently run on a stock smartphone GPU or NPU at all, because those processors lack support for the operators it uses. The NPU numbers in the paper are a projection rather than a measurement. The authors estimate that M4 on an NPU would average 0.48 s and 1.3 J, under the 0.54 s and 2.9 J they measured for task-specific models on the Pixel 7 Pro CPU, but they derive that by applying the CPU-to-NPU ratio they observed for task-specific models, not by running M4 on an NPU. Their case for a simpler accelerator rests on a separate Pixel 7 Pro measurement, where they converted 110 downloaded models to TensorFlow Lite and only 8% ran entirely on the NPU, those gaining a median speedup above 20 times over the CPU. M4 itself uses 39 operator types against the 156 that the 50 task-specific models need between them. The authors name their own limits. They write that the accuracy results come from an A100 and the Jetson board rather than from phones, that M4 underperforms task-specific models on some tasks including translation, and that a prototype assembled from off-the-shelf pre-trained models is “still highly inefficient in terms of accuracy and model parameter size”. Its backbone is Meta’s LLaMA-7B at 8-bit, with encoders taken from ImageBind and Whisper, and they note that adapters trained against one backbone stop working when the backbone is upgraded, so the design still needs a stable interface between the two. Code and benchmark are published at github.com/UbiquitousLearning/MobileFM, and the paper carries ACM copyright rather than an open license. Source: https://dl.acm.org/doi/10.1145/3636534.3649361 Read the article: https://llmobile.kavents.com/ticker/mobile-foundation-model-as-firmware/ResearchBenchmarksNPUPixelLlamahttps://llmobile.kavents.com/ticker/executorch-alpha/https://llmobile.kavents.com/ticker/executorch-alpha/Tue, 30 Apr 2024 18:00:00 +0200PyTorch released ExecuTorch alpha on April 30, 2024, the first version of its edge runtime aimed at large language models rather than small vision and speech networks. PyTorch says the work enables Llama 2 7B to run on iPhone 15 Pro, iPhone 15 Pro Max and Samsung Galaxy S22, S23 and S24 phones, with early support for Llama 3 8B. The example project shipped with the release reports 8.15 to 11.6 tok/s for the quantised 7B model, measured by PyTorch over adb on a Galaxy S22, a Galaxy S24 and a OnePlus 12. The 0.1 preview from October 2023 had no such path. Fitting a 7B model on a phone came down to quantisation. Every linear layer carries 4-bit weights in groups of 128 or 256 values, with activations quantised to 8-bit at runtime from their observed range. PyTorch measured WikiText perplexity at 9.16 for the full-precision model against 10.2 and 10.7 for the two group sizes, and notes that groups smaller than 128 were not enabled because the resulting files were still too large, since the embedding table and the weight scales remained in 32-bit floats. Support for 16-bit floats was under way. Arm, Apple and Qualcomm Technologies worked on the release, and PyTorch describes the mobile accelerator paths through Core ML, MPS, TOSA and the Qualcomm AI Stack as ongoing work rather than finished. The release notes add Snapdragon 8 Gen 3 support and 4-bit and 16-bit quantisation on the Qualcomm side, over 100 operators on Apple’s MPS backend, and a Vulkan delegate for mobile GPUs. PyTorch also credits Google for joint work on XNNPACK, the CPU library that carried the language model numbers, and says MediaTek was enabling the Llama models on its own chips. PyTorch was explicit about what alpha did not yet do. The release notes mark XNNPACK as the only finished performance path for generative models and everything else as in progress, and the announcement calls on-device LLM support early while asking the community for help building GPU and NPU delegates and expanding the quantisation schemes. Meta was already shipping the runtime elsewhere, using it for hand tracking on Quest 3, for several models on the Ray-Ban Meta smart glasses and in a beginning rollout inside Instagram. The Python package became installable from PyPI with this release. Source: https://pytorch.org/blog/executorch-alpha/ Read the article: https://llmobile.kavents.com/ticker/executorch-alpha/PyTorchLlamaQuantisationAppleQualcommhttps://llmobile.kavents.com/ticker/on-device-llmaas-context/https://llmobile.kavents.com/ticker/on-device-llmaas-context/Mon, 18 Mar 2024 15:03:23 +0100Researchers at Peking University, the Beijing University of Posts and Telecommunications and Tsinghua University published LLMS on March 18, 2024, an operating system service that keeps a separate conversation context for every app calling one shared on-device language model. Serving eight active contexts of Llama2-7B on a Jetson Orin NX, the authors measure 0.27 seconds to switch to another app’s context. They report switches up to 20 times and on average 9.7 times faster than a baseline that swaps the same chunks but quantises every one to 8 bits, modelled on the server system vLLM. The premise is one model held by the operating system instead of a copy inside each app, the way Android AICore serves Google apps, with every caller keeping its own KV cache, the stored attention state that lets a model continue a conversation without rereading it. On a Xiaomi 14 running llama.cpp, the authors measure one Llama2-7B context at the full 4k-token window at 2.02 GB, against 3.92 GB for the 4-bit weights, so a few apps holding contexts outweigh the model they all share. Letting the phone’s low-memory killer reclaim that memory means rebuilding the context by running the text through the model again, which the authors put at 22.92 seconds and 94.57 J on the same phone, close to the 90.04 J they measure for one minute of YouTube. What moves to storage is the KV cache, cut into chunks of 16 tokens, while the prompt and output text stay resident. The service quantises each chunk to 8, 4 or 2 bits according to how much attention its tokens draw from the rest of the context, holding a 50 percent average set by the system, then refills a context by reading some chunks from disk while recomputing others from the stored text and overlapping the two. The authors report twice the compression ratio of static quantisation at negligible accuracy loss, where compressing every chunk alike to 4 bits or 2 bits costs up to 59 percent and 99 percent of accuracy. Each of the three techniques carries part of that 0.27 seconds, which the authors break down as 0.42 seconds without the per-chunk compression, 0.62 seconds without the eviction order and 1.62 seconds without the read-and-recompute pipeline. The prototype is 3.5k lines of Python and C++ on Hugging Face Transformers and mllm, run on a Jetson Orin NX, a Jetson TX2 and a Xiaomi 14 with a Snapdragon 8 Gen 3, each with 8 GB of RAM, using Llama2-7B and OPT-6.7B over 72-hour traces synthesised from six datasets, and the authors state that it leaves token generation speed within 5 percent of running without it. The paper is on arXiv under a CC BY 4.0 license, and the same four authors published the work at SenSys 2026 in Saint-Malo in May 2026 as An Efficient Context Management System for On-Device LLMaaS. Charts: Yin et al. Measured on a Xiaomi 14, with the memory breakdown taken on llama.cpp. Source: https://arxiv.org/abs/2403.11805 Read the article: https://llmobile.kavents.com/ticker/on-device-llmaas-context/ResearchMemoryQuantisationLlamaAndroidhttps://llmobile.kavents.com/ticker/qualcomm-ai-hub/https://llmobile.kavents.com/ticker/qualcomm-ai-hub/Mon, 26 Feb 2024 09:00:00 +0100Qualcomm launched Qualcomm AI Hub at Mobile World Congress in Barcelona on February 26, 2024, a developer service built around a model library of more than 75 pre-optimised AI and generative AI models for Snapdragon and Qualcomm platforms. Qualcomm states that spreading each model across the NPU, CPU and GPU of its AI Engine results in 4X faster inferencing times. The second half of the service is measurement. A developer uploads a trained model, the hub compiles it for a chosen chip and runtime, then runs it on a physical Snapdragon device in Qualcomm’s cloud and returns latency, memory use and a numerical check against the original, as the documentation describes. The same measurements are published for every catalogue entry, which is why each model page names the handset the figures came from. Diagram: Qualcomm. At launch the export recipes in Qualcomm’s repository covered two runtimes, TensorFlow Lite and Qualcomm AI Engine Direct, on Android 11 and newer, in FP16, INT8 and INT4. The supported chips ran from the Snapdragon 845 through the Snapdragon 888 to the Snapdragon 8 Gen 3, with Samsung Galaxy S21 to S24, Xiaomi 12 and 13 and Google Pixel 3 to 5 handsets named as targets. Everything is driven from a Python package called qai_hub. Two of the listed models generate text, Llama-v2-7B-Chat and Baichuan-7B, both quantised to 4-bit weights and 16-bit activations. Qualcomm’s model card puts Llama-v2-7B-Chat at 3.6 GB with a maximum context of 1024 tokens, and reports it measured on a Samsung Galaxy S23 Ultra running Android 13 on a Snapdragon 8 Gen 2, on the NPU, at 117.8 ms for each additional token and 2,578.5 ms to push a full 1024-token prompt through before the first token comes back. Peak memory for the token generator is 66 to 238 MB. Qualcomm publishes the models on the AI Hub site, on GitHub and on Hugging Face, and names no price for the service, asking only for a Qualcomm ID to sign in and an API token to submit jobs. The licensing is layered. Qualcomm’s export and optimisation code is BSD 3-Clause, each model keeps the licence of the original implementation, so Llama-v2-7B-Chat stays under Meta’s terms, and the compiled assets prepared for on-device deployment carry a separate Qualcomm AI Hub Proprietary License. Source: https://aihub.qualcomm.com/mobile/models Read the article: https://llmobile.kavents.com/ticker/qualcomm-ai-hub/QualcommDeveloper toolsAndroidNPULlamahttps://llmobile.kavents.com/ticker/arm-generative-ai-on-mobile/https://llmobile.kavents.com/ticker/arm-generative-ai-on-mobile/Wed, 17 Jan 2024 15:00:00 +0100Arm showed a chat assistant running Meta’s Llama 2 7B on existing Android phones using three Cortex-A700 series CPU cores and LPDDR5x memory, at a text generation rate of 9.6 tokens per second. Gian Marco Iodice and Adnan AlSinan published the demo video and the measurement on January 17, 2024, and state that the video runs at actual speed and that everything runs locally on the device. Arm says the rate is faster than average human reading speed. Integer quantisation is what brought a 7B model into phone memory, according to Arm’s write-up. Arm puts the RAM needed for Llama 2 7B with 16-bit floating point weights at about 14 GB, and says quantising those weights to 4-bit shrinks the model four times, to roughly 4 GB. Arm warns that going down to 3-bit or 2-bit costs accuracy. Pinning each worker thread to a specific core with a thread affinity mask, rather than letting Android’s scheduler place it, improved latency by over 10 percent in Arm’s tests. Arm then wrote its own 4-bit matrix-by-vector and matrix-by-matrix routines, the two functions that dominate the work in a language model, tuned for Cortex-A700 series cores. The routines are built on two instructions the cores already carry, SDOT for dot products and SMMLA for 8-bit integer matrix multiply and accumulate. Arm reports that they cut the time to the first token by over 50 percent and raised text generation by 20 percent against the native implementation in llama.cpp, and said at the time that the routines would be available soon. Arm makes the case for the CPU on the shape of the workload rather than on peak throughput, arguing that the typical batch size and the balance of compute and bandwidth in language model inference suit a CPU, and that its programmability lets developers try new compression and quantisation schemes. The post names no NPU or GPU and offers no comparison against one. Arm puts the reach of the approach at around 99 percent of smartphones worldwide and says 70 percent of AI in third-party applications already runs on Arm CPUs. Arm later extended the same post with a second demo, a teaching assistant chatbot built on Microsoft’s Phi-3 3.8B, which Arm measures at just over 15 tokens per second on the same software optimisations. Arm names the Scalable Matrix Extension for Armv9-A as the next hardware step for these workloads. Source: https://developer.arm.com/community/arm-community-blogs/b/ai-blog/posts/generative-ai-on-mobile-on-arm-cpu Read the article: https://llmobile.kavents.com/ticker/arm-generative-ai-on-mobile/ArmAndroidllama.cppLlamaQuantisationhttps://llmobile.kavents.com/ticker/tinyllama/https://llmobile.kavents.com/ticker/tinyllama/Thu, 04 Jan 2024 18:54:00 +0100Researchers from the StatNLP group at Singapore University of Technology and Design published TinyLlama on January 4, 2024, a language model of 1.1B parameters pretrained on 3 trillion tokens. The authors set out to see how a small model behaves when it is trained on far more tokens than the compute-optimal scaling law of Hoffmann et al. recommends for its size, and they state that this is, to their knowledge, the first attempt to train a model of around 1B parameters on that much data. The corpus mixes natural language from SlimPajama with code from the StarCoder training data at a sampling ratio of roughly 7 to 3, which the authors put at about 950 billion tokens once the two are merged and the GitHub part of SlimPajama is dropped to avoid duplication. TinyLlama passes over that set for approximately three epochs, which is how the 3 trillion cumulative figure is reached. Pretraining ran on 16 A100-40G GPUs, and the project repository states that the run started on September 1, 2023 and was budgeted at 90 days. TinyLlama copies the architecture and tokenizer of Meta’s Llama 2, so it carries rotary position embeddings, RMSNorm pre-normalisation and SwiGLU activations across 22 layers with a hidden size of 2048 and a 2048-token context. It also inherits grouped-query attention, where the 32 query heads share four groups of key and value representations instead of each head holding its own, which the authors use to cut memory bandwidth and speed up inference. They report a training throughput of 24,000 tokens per second per A100-40G GPU, and measure 3,456 GPU hours to cover 300 billion tokens against 4,830 hours for Pythia-1.0B and 7,920 for MPT-1.3B in their own comparison. They attribute that speed to existing open-source components rather than anything new, naming Fully Sharded Data Parallel for spreading training across nodes, FlashAttention-2, fused implementations of layer normalisation, the cross entropy loss and the rotary position embedding, and the fused SwiGLU module from xFormers. Across seven zero-shot commonsense reasoning benchmarks the authors’ own table puts TinyLlama at an average of 52.99, ahead of OPT-1.3B at 51.44, Pythia-1.4B at 51.33 and Pythia-1.0B at 48.30, with the widest margin on HellaSwag at 59.20 against 53.65 for OPT-1.3B. It does not lead every column, and Pythia-1.4B stays ahead on BoolQ at 63.27 against 57.83. On the InstructEval suite TinyLlama averages 19.87 against 17.72 for Pythia-1.4B and 16.95 for OPT-1.3B, with most of that difference coming from HumanEval, where it scores 9.15 against 4.27. The authors put the device case briefly, writing in the paper that the compact architecture “can enable end-user applications on mobile devices”, and the repository describes the model as suited to applications with a restricted computation and memory footprint. It puts the 4-bit quantised weights at 637 MB and names offline real-time machine translation and drafting tokens that a larger model then checks as the uses it has in mind. Neither the paper nor the repository reports a measurement on a phone. Weights and all intermediate checkpoints are on Hugging Face under Apache 2.0, with the pre-training code in the repository. Source: https://arxiv.org/abs/2401.02385 Read the article: https://llmobile.kavents.com/ticker/tinyllama/LlamaResearchOpen weightsOpen sourcehttps://llmobile.kavents.com/ticker/hf-swift-transformers/https://llmobile.kavents.com/ticker/hf-swift-transformers/Tue, 08 Aug 2023 16:15:00 +0200Hugging Face published swift-transformers on August 8, 2023, the date of the repository’s first tag, a Swift package that gives iOS and Mac developers a transformers-like API for running language models on the device through Core ML. Pedro Cuenca introduced it as an in-development library built around four modules, Tokenizers, Hub, Generation and Models. The Tokenizers module ports the normalisation and pre-processing rules that the Python tokenizers library keeps as configuration files next to a model on the Hub, so an app does not hardcode those decisions per model and can swap one checkpoint for another. It covered BPE at release, the family used by the GPT models, Falcon and Llama, with Unigram and WordPiece listed as later work. The Hub module downloads those configuration files and the model metadata, and Generation implements greedy decoding and top-k sampling. The Models module wraps a converted Core ML package behind a LanguageModel type, and Core ML then decides whether to run it on the CPU, the GPU or the Neural Engine. Cuenca writes that flexible input shapes run on the CPU only, which is why Hugging Face’s exporters conversion tool switched to fixed sequence lengths, and that the Core ML path cached no attention keys and values at that point, both of which cap how long a context an app can work with. Hugging Face states it tested the package with Llama 2, Falcon, StarCoder and the GPT family including GPT2, distilgpt, GPT-NeoX and GPT-J, and that encoder-decoder models such as T5 and Flan were not supported. The release came alongside swift-chat, a Mac app that loads a Core ML model and doubles as a model tester, plus converted weights for Llama 2 7B and Falcon 7B on the Hub. The package is licensed under Apache 2.0. Screenshot: Hugging Face. The demo app exposes the generation parameters and reports throughput. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: Hugging Face. swift-chat generating a haiku from Llama 2 7B on an M1 MacBook Pro. Open the Hugging Face post Source: https://github.com/huggingface/swift-transformers Read the article: https://llmobile.kavents.com/ticker/hf-swift-transformers/Developer toolsiOSApple SiliconOpen sourceLlamahttps://llmobile.kavents.com/ticker/qualcomm-hybrid-ai-whitepaper/https://llmobile.kavents.com/ticker/qualcomm-hybrid-ai-whitepaper/Mon, 15 May 2023 18:00:00 +0200Qualcomm published a whitepaper, The future of AI is hybrid, in May 2023 arguing that generative AI inference has to be distributed between the cloud and edge devices rather than run in data centres alone. The company’s case rests on cost. It cites a Reuters report estimating that a generative AI search costs 10 times more per query than traditional search, and a Morgan Stanley note from February 2023 saying that with more than 10 billion search queries a day, even a small share moving to language models could add multiple billions of dollars a year. On what can run locally, Qualcomm writes that models of 1 to 10 billion parameters, assuming INT4 weights, cover capabilities including text-to-image, dialog, image understanding and video understanding. It states that models with more than 1 billion parameters, such as Stable Diffusion, already run on phones with performance and accuracy similar to the cloud equivalent, and that models of 10 billion parameters or more are slated to run on devices in the near future. The chart in the paper puts the phone and laptop reach at about 10 billion parameters for 2023 and extends it further for 2024, while the cloud range spans the whole axis to 1000 billion. Chart: Qualcomm. The company's own figure for which model sizes it expects devices to handle. The paper sets out three ways of splitting a workload. In the device-centric option the device runs most of the inference and an on-device neural network or rules-based arbiter decides when a query needs the cloud, with Qualcomm expecting laptops to run models of up to tens of billions of parameters. In the device-sensing option, speech recognition such as Whisper, computer vision and text-to-speech run on the phone while a cloud model does the language work, which the company argues cuts both compute and bandwidth. The third option has device and cloud work on the same response through speculative decoding, a technique published by Google researchers in 2022. Qualcomm’s example has a draft model 7 to 10 times smaller than the target model generate four tokens on the device, which the cloud then checks by running the target model once across the four candidates, since a model like GPT-3 has to read all 175 billion parameters from memory to produce a single token. The company says its own early experiments and published results show 2 to 3 of the 4 draft tokens being accepted on average. The only on-device demonstration the paper cites is Qualcomm’s own Stable Diffusion run on an Android phone from February 2023. Its other device claims are projections, including text-to-image models generating 3D textures on smartphones within a year and a 13 billion parameter LLaMA model reaching edge devices to voice avatars in XR. On energy the document argues only that edge devices run generative AI at a fraction of the cloud’s consumption once data transport is counted, without giving a figure. The whitepaper is Part I of two, announced in a Qualcomm blog post by Ziad Asghar and Jilei Hou, with Part II covering the company’s own products. Source: https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/Whitepaper-The-future-of-AI-is-hybrid-Part-1-Unlocking-the-generative-AI-future-with-on-device-and-hybrid-AI.pdf Read the article: https://llmobile.kavents.com/ticker/qualcomm-hybrid-ai-whitepaper/QualcommQuantisationMemoryLlamahttps://llmobile.kavents.com/ticker/sherpa/https://llmobile.kavents.com/ticker/sherpa/Mon, 27 Mar 2023 20:46:00 +0200Bip-Rep published Sherpa on March 27, 2023, a Flutter app that runs llama.cpp on the phone itself, so the chat works with no network connection. The README asks users to bring their own weights, Meta’s 7B LLaMA converted to a ggml model file, which the app loads through a file picker, and version 2.1.0 added Vicuna support and macOS builds next to the Android and Windows ones. The project publishes no tokens per second figure and states no minimum memory, and the only device it names is a OnePlus 7 with 8 GB of RAM in the demo video below, which the README says was recorded at real speed. The code is MIT licensed. Development stopped three weeks after it started. All 43 commits fall between March 27 and April 14, 2023, the last of the seven releases went out on April 14, 2023, and the llama.cpp fork the app builds against was last touched on April 12, 2023, four months before llama.cpp moved to the GGUF format. Sherpa reads the older ggml files rather than the GGUF models that have been standard since, and requests for GGUF support have gone unanswered, with 23 of the 29 issues ever filed still open. A contributor answered one of them by writing that the original developers had abandoned the project, and pointed to his own fork, Maid. ▶A working chatGPT running on a mobile using llama.cppLoading connects your browser to www.youtube-nocookie.com, which may process your IP address and use cookies. Load contentOpen externallyVideo: tibzejoker. Source: https://github.com/Bip-Rep/sherpa Read the article: https://llmobile.kavents.com/ticker/sherpa/llama.cppAndroidLlamaOpen source