Llama
22 updates on Llama.
torchchat runs Llama 3 8B on a Galaxy S23 and iPhone at more than 8 tok/s
PyTorch's demonstration codebase exports Llama models to ExecuTorch and ships iOS and Android sample apps for running them on a phone.
BUPT proposes one 9.2B model in the OS that all apps call through adapters
A BUPT team proposes that the phone OS ship one 9.2B multimodal model all apps call through small adapters, and matched app models on 85% of 50 datasets.
ExecuTorch alpha runs Llama 2 7B on iPhone 15 Pro and Galaxy phones
PyTorch's edge runtime brought 4-bit Llama 2 7B to iPhone and Galaxy handsets, added early Llama 3 8B support and leaned on Apple, Arm and Qualcomm.
One shared on-device LLM keeps a context per app and switches in 0.27 seconds
Peking University, BUPT and Tsinghua researchers compress and swap each app conversation context in 16-token chunks, switching up to 20 times faster.
Qualcomm AI Hub opens 75 pre-optimised models and profiling on hosted Snapdragon phones
Qualcomm opened a library of more than 75 models tuned for Snapdragon, with compilation and profiling on real phones in its cloud and two 7B chat models listed.
Arm runs Llama 2 7B at 9.6 tokens per second on three mobile CPU cores
Arm demonstrated Llama 2 7B generating 9.6 tokens per second on three Cortex-A700 CPU cores, using int4 kernels it says beat llama.cpp by 20 percent.
TinyLlama pretrains a 1.1B model on 3 trillion tokens
Singapore University of Technology and Design trained a 1.1B model on 3 trillion tokens with 16 A100-40G GPUs and released it under Apache 2.0.
Hugging Face publishes swift-transformers for Core ML models in Swift apps
A Swift package with tokenizers, Hub downloads and a Core ML generation loop, demonstrated with Llama 2, Falcon, StarCoder and GPT models.
Qualcomm argues for hybrid AI, citing 10 times the cost per generative AI search query
Qualcomm argues that cloud-only inference cannot scale, and puts models of 1B to 10B parameters on phones and laptops at INT4.
Sherpa runs LLaMA on an Android phone through a Flutter chat app
The Flutter app loads a ggml model file off the phone and chats with it offline, and development stopped three weeks after it started.