On-device
Georgi Gerganov publishes llama.cpp, LLM inference in plain C and C++
The dependency-free C and C++ inference engine supports 1.5-bit to 8-bit quantisation and treats Apple silicon as a first-class target.
Qualcomm runs Stable Diffusion on an Android phone for the first time
Qualcomm AI Research generated 512x512 images in under 15 seconds on a Snapdragon 8 Gen 2 phone after quantising the model from FP32 to INT8.
George Hotz starts tinygrad, a framework that ports to an accelerator in about 25 ops
The tiny corp framework caps its repository at 26,500 lines, ships Metal, Adreno and WebGPU backends, and runs openpilot on a Snapdragon 845 GPU.
ONNX Runtime 1.5 adds a mobile build that ships only the operators a model uses
The release introduced a reduced-size runtime for phones and the ORT model format, the base for the NNAPI, Core ML, QNN and XNNPACK mobile packages.
MobileBERT runs in 62 ms on a Pixel 4 with 25.3M parameters
Carnegie Mellon and Google Brain distilled BERT into a 25.3M-parameter model that answers in 62 ms on a Pixel 4 and scores 77.7 on GLUE.
Hugging Face's DistilBERT is 40% smaller than BERT and 71% faster on an iPhone
The 66M-parameter distilled BERT keeps 97% of its teacher on GLUE and weighs 207 MB inside a question-answering app on an iPhone 7 Plus.