Quantisation
38 updates on Quantisation.
OpenBMB releases MiniCPM5-2B for local deployment
A 2.5-billion-parameter dense model with a 131,072-token context, released under Apache-2.0 with GGUF, MLX and LiteRT-LM builds.
Artificial Analysis benchmarks 33 local models on an iPhone 17 Pro
A benchmark of quantised small models on an iPhone 17 Pro reports intelligence scores, generation times and peak memory between 0.4 and 6.9 GB.
Apertus Mini distils an open-data 8B into 0.5B, 1.5B and 4B models
Swiss AI distilled its fully open Apertus 8B into 0.5B, 1.5B and 4B models on 1.7T tokens, with 3-bit to 6-bit MLX builds for Apple devices.
Meta AI builds MobileMoE, 5.3B parameters with 0.9B active per token
Meta AI trained three on-device mixture-of-experts models that store 1.3B to 5.3B parameters and run 272M to 922M of them per token.
Quant.npu runs 4-bit LLMs up to 15.1 percent faster on a Qualcomm NPU
Eight authors report a fully static integer quantisation method that runs 4-bit models up to 15.1 percent faster on a Qualcomm SM8650 NPU.
Tencent open-sources a 440 MB offline translation model for phones
Hy-MT1.5-1.8B-1.25bit compresses a 1.8-billion-parameter translation model from 3.3 GB to 440 MB and runs offline on a phone. Weights and an Android demo are public.
Alibaba adds 0.8B and 2B sizes to Qwen3.5, with 262K context and a vision encoder
Alibaba released Qwen3.5-0.8B and Qwen3.5-2B, dense vision-language models with a 262,144-token context, Apache 2.0 weights and 4-bit MNN builds.
Meta's MobileLLM-Pro runs a 128k context from a 590 MB 4-bit build
Meta Reality Labs released a 1.08B on-device model with a 128k context window, measured at 33.6 tok/s decode on a Galaxy S25 CPU.
Apple puts the cost of 2-bit compression at 3.4 MMLU points
Apple measures its on-device model at 67.8 MMLU in 16 bits and 64.4 after compression to 2 bits per weight, and details the distillation pipeline behind it.
Apple opens its on-device model to all apps with the Foundation Models framework
Any app can call the roughly 3-billion-parameter on-device model from Swift, offline and free of charge, with guided generation and tool calling.
Vijay Janapa Reddi argues edge generative AI needs models under 1B parameters
In ACM Queue, Vijay Janapa Reddi argues generative AI reaches edge devices only through small deployable models, with sub-1B ones the viable case.
D2MoE picks a bit-width per token, 1.39 times the throughput at up to 53 percent less memory
A MobiCom 2025 paper routes every token to an expert and a bit-width, reporting up to 1.39 times the throughput of EdgeMoE at up to 53 percent less memory.