https://llmobile.kavents.com/tags/webgpu/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/flower-intelligence/https://llmobile.kavents.com/ticker/flower-intelligence/Wed, 12 Mar 2025 10:00:00 +0100Flower Labs released Flower Intelligence on March 12, 2025 as a preview, an open source inference library that runs a language model on the user’s own device and only moves the work to a remote service if the app and the user allow it. The company shipped SDKs for Swift and TypeScript, and the TypeScript build defaults to Llama 3.2 1B in 16-bit precision. The library picks a local engine from the environment it finds itself in. The TypeScript package depends on MLC’s WebLLM, which executes models in the browser through WebGPU, and on Hugging Face transformers.js, which it uses when the code runs under Node.js rather than in a page. The Swift package is built on MLX Swift and Hugging Face swift-transformers, and declares iOS 16 and macOS 14 as its minimum platforms, so local inference on Apple hardware goes through MLX. Flower Labs’ model reference covers Llama 3.2 at 1B and 3B, Llama 3.1 8B, SmolLM2 at 135M, 360M and 1.7B, and DeepSeek R1 Distill Llama 8B, most of them as 4-bit builds. Which of those run on device depends on the backend. Node.js is limited to the 1B and SmolLM2 entries, while the 8B models run only in the browser and on the MLX Swift path, and Llama 3.2 3B in 16-bit is MLX Swift only. Remote handoff is off by default. An application has to set the remoteHandoff property and supply an API key before the library will contact Flower Confidential Remote Compute, and even with both in place a chat call tries a local engine first and retries remotely only when no local engine matches the model or the local run fails. Individual requests can be pinned with forceLocal or forceRemote. Flower Labs describes the remote service as a private extension of the device that costs the developer no extra work, and named Mozilla Thunderbird as an early adopter building its Thunderbird Assist feature on the library. Ryan Sipes, Managing Director for Mozilla Thunderbird, tied that choice to the mail client’s user base in the announcement. Our 20 million users expect data privacy from every feature we build. Both SDKs are published under the Apache 2.0 license, the TypeScript one as @flwr/flwr on npm and the Swift one as a package in the Flower repository. Flower Labs called the release inference-only and listed fine-tuning, RAG and pre-training as later steps, with the remote compute service available on application for early access. Source: https://flower.ai/blog/2025-03-12-flower-intelligence Read the article: https://llmobile.kavents.com/ticker/flower-intelligence/Open sourceDeveloper toolsLlamaWebGPUApple Siliconhttps://llmobile.kavents.com/ticker/smollm2/https://llmobile.kavents.com/ticker/smollm2/Thu, 31 Oct 2024 19:30:00 +0100Hugging Face released SmolLM2 on October 31, 2024, a second generation of its small language models at 135M, 360M and 1.7B parameters. The company trained the 1.7B model on 11 trillion tokens, against 1T for the first SmolLM at the same size, and its model cards put the 360M model at 4T tokens and the 135M at 2T. The mixture combines FineWeb-Edu, DCLM and The Stack with datasets Hugging Face curated for this run, which its paper from February 2025 names as FineMath for mathematics, Stack-Edu for code and SmolTalk for instruction tuning. That paper states the 1.7B run used a multi-stage schedule that held the highest quality maths and code data back for the final tenth of training, from 10T to 11T tokens, while the 360M and 135M models trained in a single stage on a mix kept consistent throughout. Hugging Face fine-tuned the instruct versions on SmolTalk and then tuned them further on ranked answer pairs from UltraFeedback. Hugging Face reports the 1.7B base model at 60.5 on ARC against 49.2 for Llama 3.2 1B and 58.5 for Qwen2.5 1.5B, 36.7 on TriviaQA against 28.1 and 20.9, and 19.4 on MMLU-Pro against 11.7 and 13.7, all run zero-shot with its own lighteval harness and printed in the model card. The instruct version scores 56.7 on the IFEval instruction-following test against 53.5 for Llama 3.2 1B Instruct and 47.4 for Qwen2.5 1.5B Instruct, while Qwen2.5 stays ahead on the MT-Bench chat test at 6.52 to 6.13. At the smaller sizes Hugging Face compares the 360M model with Qwen2.5 0.5B, reporting 53.0 against 45.4 on ARC and 16.9 against 4.3 on TriviaQA, and publishes no third-party comparison for the 135M model. Table: Hugging Face's own figures for the base models. The weight files run 269 MB for the 135M model, 724 MB for the 360M and 3.42 GB for the 1.7B in bfloat16, and Hugging Face uploaded 4-bit ONNX builds the same day at 118 MB, 273 MB and 1.11 GB, plus a 4-bit GGUF of the 1.7B instruct model at 1.06 GB. The repository says the models run locally through llama.cpp, MLX, MLC and transformers.js. All six checkpoints carry an 8k context window and the Apache 2.0 license. Update, November 27, 2024. Hugging Face published a browser chat demo that runs the 1.7B instruct model through transformers.js and ONNX Runtime Web on WebGPU, with no server call after the download. Its local inference guide adds runnable code for llama.cpp, MLX and MLC, and lists the models on Ollama and in the PocketPal-AI phone app. Source: https://github.com/huggingface/smollm Read the article: https://llmobile.kavents.com/ticker/smollm2/Open weightsBenchmarksOpen sourcellama.cppWebGPUhttps://llmobile.kavents.com/ticker/smollm/https://llmobile.kavents.com/ticker/smollm/Tue, 16 Jul 2024 16:00:00 +0200Hugging Face released SmolLM on July 16, 2024, a family of three base models at 135M, 360M and 1.7B parameters. The team says the models are designed to be small enough to run locally across a range of hardware, from smartphones to laptops, and cites the iPhone 15 with 6 GB of DRAM and the iPhone 15 Pro with 8 GB as reference points. Hugging Face trained the 135M and 360M models on 600B tokens each and the 1.7B model on 1T tokens. The substance is the training data, released as SmolLM-Corpus. Cosmopedia v2 contributes 28B tokens of synthetic textbooks, stories and articles generated by Mixtral-8x7B-Instruct, 39 million documents built from a predefined list of 34,000 topics taken from the BISAC book classification instead of from unsupervised clustering. FineWeb-Edu contributes 220B deduplicated tokens of educational web pages, pulled out of FineWeb by a quality classifier trained on annotations from Llama3-70B-Instruct. Python-Edu applies the same idea to code, keeping 4B tokens of the 40B Python tokens in The Stack that scored 4 or higher on an educational code classifier. On Hugging Face’s own evaluation, run with the same lighteval setup for every model it could download, SmolLM-135M averages 39.50 across MMLU, TriviaQA, ARC, PIQA, HellaSwag, OpenBookQA and Winogrande, against 32.75 for GPT2-137M and 32.49 for Pythia-160M. SmolLM-360M averages 44.44 against 39.60 for Qwen2-500M and 39.32 for Qwen1.5-500M, and SmolLM-1.7B averages 51.97 against 50.36 for Microsoft’s Phi-1.5 and 47.98 for Qwen2-1.5B. Meta’s MobileLLM is not publicly available, so Hugging Face took its per-benchmark numbers from the paper and reports no average for those rows. Table: Hugging Face's own figures, evaluated with the same setup for every model except MobileLLM. Hugging Face also benchmarked the memory footprint of each size. SmolLM-135M takes 269.03 MB in bf16, 162.87 MB at int8 and 109.78 MB at int4. SmolLM-360M takes 723.65 MB, 409.07 MB and 251.79 MB across the same three precisions, and SmolLM-1.7B takes 3422.76 MB, 1812.14 MB and 1006.84 MB. Table: Hugging Face. The weights are on Hugging Face under Apache-2.0, with ONNX checkpoints published beside the transformers ones and a GGUF build for llama.cpp announced as planned. Hugging Face also put up browser demos of the instruction-tuned 135M and 360M models that run client-side through WebGPU, with no server call. The corpus is released separately under ODC-By. Source: https://huggingface.co/blog/smollm Read the article: https://llmobile.kavents.com/ticker/smollm/Hugging FaceOpen weightsBenchmarksMemoryWebGPUhttps://llmobile.kavents.com/ticker/mlc-llm-consumer-devices/https://llmobile.kavents.com/ticker/mlc-llm-consumer-devices/Mon, 01 May 2023 18:00:00 +0200The MLC project released MLC LLM on May 1, 2023, a compiler and runtime that deploys language models natively on phones, browsers and consumer hardware rather than serving them from a data centre. The team pointed users to a TestFlight build to try the chat app on iOS. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: MLC. The iOS app generating a reply on the phone. Open the MLC post MLC LLM compiles models through the Apache TVM Unity stack and maps them to the Vulkan and Metal graphics APIs, which the team says covers the majority of consumer platforms. Targets named at launch include Windows, Linux, macOS and iOS, plus AMD, NVIDIA, Intel and Apple GPUs and a Steam Deck. A companion project, WebLLM, offloads the same models into web browsers through WebGPU. Three weeks later the team added RedPajama-3B across the same set of targets, with a browser demo and a command line tool for desktop machines. The code is on GitHub under the Apache 2.0 license. The project now exposes an OpenAI-compatible API through a REST server as well as Python, JavaScript, iOS and Android bindings, and covers Android GPUs via OpenCL alongside CUDA and ROCm on the desktop. Source: https://blog.mlc.ai/2023/05/01/bringing-accelerated-llm-to-consumer-hardware Read the article: https://llmobile.kavents.com/ticker/mlc-llm-consumer-devices/MLC LLMiOSiPhoneOpen sourceWebGPUhttps://llmobile.kavents.com/ticker/tinygrad/https://llmobile.kavents.com/ticker/tinygrad/Sun, 18 Oct 2020 07:57:00 +0200George Hotz pushed the first commit of tinygrad on October 18, 2020, with the message “start tinygrad”, according to the repository’s commit history. The README describes an end-to-end deep learning stack with a tensor library, an autograd, a compiler that fuses and lowers kernels, and a JIT, positioned between PyTorch and karpathy/micrograd. It also states that an accelerator port needs a total of about 25 low level ops. tiny corp, the company Hotz founded, maintains it. The README lists backends for OpenCL, CPU, Metal, CUDA, AMD, NVIDIA, QCOM and WebGPU. The runtime documentation names the requirements for each one, Adreno 6xx series GPUs for the QCOM backend and M1 or newer Macs for Metal, with Metal 3.0 needed for bfloat support. The QCOM backend, which drives the Qualcomm GPU through the Adreno kernel driver interface instead of OpenCL, entered the repository on September 2, 2024, and the repository also carries a backend for Qualcomm’s Hexagon DSP in ops_dsp.py. tiny corp states on its site that tinygrad is used in openpilot to run the driving model on the Snapdragon 845 GPU, where it replaces Qualcomm’s SNPE. The company says its framework is faster than SNPE, loads ONNX files, supports training and allows attention, which SNPE does not because it only allows fixed weights. The repository’s test workflow compiles an openpilot model with an image pitch alignment set to match the Adreno 630, the GPU in the Snapdragon 845. The same workflow enforces the project’s size, failing the build when the repository goes over 26,500 lines as counted by sz.py. The repository ships a language model tool in tinygrad/llm that loads GGUF files, serves an OpenAI compatible API and benchmarks tokens per second, with a built-in model list that starts at quantised Llama 3.2 1B and Qwen3 0.6B. tinygrad is published under the MIT license, and the project recommends installing from source rather than from PyPI. Diagram: tinygrad. Source: https://github.com/tinygrad/tinygrad Read the article: https://llmobile.kavents.com/ticker/tinygrad/Open sourceDeveloper toolsQualcommApple SiliconWebGPU