Apertus Mini distils an open-data 8B into 0.5B, 1.5B and 4B models

Researchers working in the Swiss AI Initiative published a technical report on May 27, 2026 describing three language models distilled from Apertus 8B, at 0.5B, 1.5B and 4B parameters. The authors are based at EPFL, ETH Zurich and ISTA, and the models sit on Hugging Face as the Apertus Mini collection. They inherit what the project calls a fully open and compliant release, meaning published weights, published data pipeline and published recipe, with pre-training restricted to openly available data that honours robots.txt opt-outs. The model card lists 1,811 natively supported languages.

Each instruction-tuned model also ships in MLX format at 3, 4 and 6 bits, which the authors aimed at Apple devices and at memory-limited mobile and edge deployment. The 4-bit 4B weights are a 2.59 GB file against 7.65 GB for the same model in bfloat16, and the 1.5B and 0.5B come to 1.08 GB and 0.31 GB. The authors build the quantised copies with quantisation-aware distillation, a short recovery run that retrains the compressed weights against the 8B model’s own output distribution, and report 90% to 104% of the unquantised few-shot accuracy on ARC, HellaSwag, MMLU and WinoGrande. Separate FP8 and NVFP4 checkpoints target NVIDIA GPUs instead.

The base model table averages ARC, HellaSwag, WinoGrande, XNLI, XCOPA and PIQA, and puts the three sizes at 51.79, 56.66 and 61.53 against 64.96 for the 8B teacher. The same table gives the Qwen3 base models 52.23, 57.51 and 62.14 at comparable sizes, so the distilled models land just under them on the average while leading on the two cross-lingual sets, 45.03 against 43.00 on XNLI and 63.82 against 61.82 on XCOPA at 4B. On a multilingual average of MMLU, TruthfulQA, ARC, instruction following and LogiQA for the instruction-tuned models, the authors report 0.318, 0.382 and 0.473 against 0.401, 0.457 and 0.521 for Qwen3 and 0.497 for Gemma 3 4B, and write that the family is weaker at instruction following and maths.

Each model saw 1.7T tokens taken from the final and most heavily filtered phase of the original Apertus data, with no new sources added, packed into 4,096-token sequences and trained on 64 GH200 GPUs at the Swiss National Supercomputing Centre. The cost table puts the 4B run at 2.0E22 FLOPs and the single pass that produced the teacher’s logits for all three students at 1.4E22, against 3.7E23 FLOPs for the original Apertus 8B pre-training on 15T tokens, which the authors describe as making the whole family cost under 12% of the teacher. Swiss AI released 24 checkpoints in total, base, instruction-tuned and quantised, under the Apache 2.0 licence with an acceptable use policy attached, along with the training, post-training and quantisation code.