ONNX Runtime 1.5 adds a mobile build that ships only the operators a model uses

Microsoft released ONNX Runtime 1.5 on September 29, 2020 with a preview of ORT for Mobile Platforms, described in the release notes as minimising build size for mobile and embedded devices. The same release added a reduced operator kernel build, which compiles the runtime with only the operators the target models actually use, and the ORT model format that those reduced builds require. The format documentation records version 1 of the ORT format as introduced in 1.5.

Workflow diagram running from PyTorch, TensorFlow and ML.NET through conversion to ONNX format, then building an application with the ONNX Runtime Mobile package, measuring performance, and optimising with an ORT format model and a minimal or custom runtime package — Diagram: ONNX Runtime documentation.

The pre-built mobile packages, onnxruntime-android for Java, C and C++ and onnxruntime-c and onnxruntime-objc for iOS, carry the full operator set and load .onnx files directly, while a custom build takes models converted to the .ort format. Microsoft’s mobile guide puts numbers on the difference for version 1.18.0, where a custom build supporting the operators of a ResNet50 model produces an Android AAR of 7,532,309 bytes against 24,415,212 bytes for the pre-built package, and an arm64-v8a libonnxruntime.so of 3,962,832 bytes against 16,276,832 bytes.

Accelerators are called execution providers. The NNAPI provider requires Android 8.1 or higher and is recommended on Android 9 or higher, the Core ML provider requires iOS 13 or macOS 10.15 and higher, and the XNNPACK provider covers floating-point CPU work on both platforms. Microsoft’s guidance is to start on the CPU provider for a quantised model and on XNNPACK otherwise, then try NNAPI or Core ML if that misses the performance target. The QNN provider reaches Qualcomm Snapdragon chips on Android and Windows through the Qualcomm AI Engine Direct SDK, with pre-built packages for Windows only and an HTP backend that takes quantised models only.

Quantisation in ONNX Runtime is 8-bit linear quantisation, offered as dynamic and static post-training quantisation in two representations, QOperator with dedicated quantised operators and QDQ with QuantizeLinear and DequantizeLinear pairs inserted around the originals. The mobile guide states that moving 32-bit weights to 8 bits cuts model size by roughly a factor of 4. The tooling also does block-wise weight-only quantisation to int4 and uint4 for MatMul and Gather nodes, with RTN, HQQ and GPTQ algorithms, and models with GatherBlockQuantized nodes need ONNX Runtime 1.20 or later to run.

Generative models run through the separate onnxruntime-genai package, which implements the generation loop, KV cache management, search and sampling around the runtime. Its support matrix lists Android as supported and iOS as on the roadmap, with CPU, QNN, WebGPU, CUDA, DirectML and OpenVINO among the acceleration options. Microsoft reported in April 2024 that RTN int4 quantisation shrinks Phi-3 Mini enough to run on a Samsung Galaxy S21 at what it called a moderate speed, and recommends the int4_accuracy_level=4 variant on mobile. Both projects are published under the MIT license.