On-device
Stability AI releases StableLM Zephyr 3B for edge devices
Stability AI tuned a 3B chat model with direct preference optimisation for edge devices and reports an MT-Bench score of 6.64.
Gemini Nano ships on the Pixel 8 Pro and Android gets AICore
Google put Gemini Nano on the Pixel 8 Pro for Recorder summaries and Gboard Smart Reply, and introduced AICore as the Android service behind it.
Apple publishes MLX, where CPU and GPU share arrays without copies
Apple machine learning research released an array framework for Apple silicon with a unified memory model, lazy evaluation and Swift bindings for iOS.
LLM.swift wraps llama.cpp for on-device text generation in Swift apps
A Swift package that wraps llama.cpp so iOS, macOS, watchOS, tvOS and visionOS apps can load a GGUF file and generate text on the device.
Snapdragon 8 Gen 3 targets 10-billion-parameter models on device
Qualcomm says the new flagship runs generative models with up to 10 billion parameters on device and reaches up to 20 tokens per second for LLMs.
MIT HAN Lab publishes TinyChatEngine for 4-bit LLMs on laptops and Raspberry Pi
The from-scratch C and C++ engine from Song Han's group at MIT runs 4-bit LLaMA2-7B on x86, Apple silicon, Raspberry Pi and Nvidia GPUs.
Hugging Face publishes swift-transformers for Core ML models in Swift apps
A Swift package with tokenizers, Hub downloads and a Core ML generation loop, demonstrated with Llama 2, Falcon, StarCoder and GPT models.
LLMFarm runs llama.cpp models offline on iOS and macOS
The iOS and macOS app loads a quantised model file off the device and chats with it offline, with 3B the practical size on an iPhone.
RWKV trains like a transformer and runs with constant memory per token
The paper releases pretrained RNN weights from 169M to 14B parameters and reports constant time and memory per token during inference.
Qualcomm argues for hybrid AI, citing 10 times the cost per generative AI search query
Qualcomm argues that cloud-only inference cannot scale, and puts models of 1B to 10B parameters on phones and laptops at INT4.
MLC LLM brings local language models to iPhone, browsers and consumer GPUs
A compiler stack built on Apache TVM deploys chat models natively to iOS, browsers and consumer GPUs, with an iPhone build handed out through TestFlight.
Sherpa runs LLaMA on an Android phone through a Flutter chat app
The Flutter app loads a ggml model file off the phone and chats with it offline, and development stopped three weeks after it started.