https://llmobile.kavents.com/tags/distillation/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/online-sdft/https://llmobile.kavents.com/ticker/online-sdft/Tue, 18 Aug 2026 16:57:46 +0200I-Ju Lin and Zhang-Wei Hong published Online-SDFT, a method that keeps fine-tuning a small language model on a phone from the user’s own stream of decisions. On a synthetic notification-routing benchmark with Liquid AI’s LFM2.5-230M, the authors report 70.28% of decisions matching a hidden user preference against 52.78% for the strongest baseline they tested, and cumulative regret of 44.76 against 105.26, over 720 decisions in three paired streams. They state that this comparison ran on Apple MPS rather than on a phone, and that the confidence intervals are nominal across only three streams. Self-distillation here means the model teaches itself. The model picks an action first, the phone later observes what happened, and the same model is then run again with that outcome added to its input. Its better-informed answer becomes the training target for the version that saw only the original context, so no second teacher model, no human label and no reward score is needed. The authors keep the 230M base frozen and train a LoRA adapter of 172,032 parameters across 48 tensors in six attention layers, with the teacher pass running with the adapter switched off so student updates cannot move the teacher. Feeding one interaction at a time into so small an adapter makes it noisy and prone to forgetting, so the authors keep a 64-row buffer of recent lessons and sample each batch from it. Their ablation puts that buffer ahead of the exploration schedule as the load-bearing part, with accuracy dropping from 70.28% to 39.58% and regret climbing from 44.76 to 134.90 once replay is removed, while dropping exploration instead leaves accuracy at 69.44% and regret at 62.00. Diagram: I-Ju Lin and Zhang-Wei Hong. The repository carries an Android proof of concept in which LFM2.5-230M decides and ONNX Runtime Training performs the adapter update on the device, with replay and checkpoints in app-private storage that survive a restart. Exporting the model still happens on a Linux host, and the deployment guide asks for Android 12 or newer on ARM64 with 12 GB of RAM at minimum, 16 GB preferred, and 8 GB of free storage, because the training graph is FP32. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: I-Ju Lin and Zhang-Wei Hong. Watch on the project page The authors call that graph a correctness implementation rather than a claim of practical background training on low-memory phones, and state that they did not measure Android latency, peak memory, battery use or thermal behaviour, and that the benchmark uses synthetic streams rather than real users. The code, the Android app and a Colab notebook are on GitHub under the MIT license. No paper or preprint accompanies the release. Source: https://lin826.github.io/SLM-Online-SDFT/ Read the article: https://llmobile.kavents.com/ticker/online-sdft/ResearchDistillationAndroidLFMOpen sourcehttps://llmobile.kavents.com/ticker/apertus-mini/https://llmobile.kavents.com/ticker/apertus-mini/Wed, 27 May 2026 23:40:00 +0200Researchers working in the Swiss AI Initiative published a technical report on May 27, 2026 describing three language models distilled from Apertus 8B, at 0.5B, 1.5B and 4B parameters. The authors are based at EPFL, ETH Zurich and ISTA, and the models sit on Hugging Face as the Apertus Mini collection. They inherit what the project calls a fully open and compliant release, meaning published weights, published data pipeline and published recipe, with pre-training restricted to openly available data that honours robots.txt opt-outs. The model card lists 1,811 natively supported languages. Each instruction-tuned model also ships in MLX format at 3, 4 and 6 bits, which the authors aimed at Apple devices and at memory-limited mobile and edge deployment. The 4-bit 4B weights are a 2.59 GB file against 7.65 GB for the same model in bfloat16, and the 1.5B and 0.5B come to 1.08 GB and 0.31 GB. The authors build the quantised copies with quantisation-aware distillation, a short recovery run that retrains the compressed weights against the 8B model’s own output distribution, and report 90% to 104% of the unquantised few-shot accuracy on ARC, HellaSwag, MMLU and WinoGrande. Separate FP8 and NVFP4 checkpoints target NVIDIA GPUs instead. The base model table averages ARC, HellaSwag, WinoGrande, XNLI, XCOPA and PIQA, and puts the three sizes at 51.79, 56.66 and 61.53 against 64.96 for the 8B teacher. The same table gives the Qwen3 base models 52.23, 57.51 and 62.14 at comparable sizes, so the distilled models land just under them on the average while leading on the two cross-lingual sets, 45.03 against 43.00 on XNLI and 63.82 against 61.82 on XCOPA at 4B. On a multilingual average of MMLU, TruthfulQA, ARC, instruction following and LogiQA for the instruction-tuned models, the authors report 0.318, 0.382 and 0.473 against 0.401, 0.457 and 0.521 for Qwen3 and 0.497 for Gemma 3 4B, and write that the family is weaker at instruction following and maths. Each model saw 1.7T tokens taken from the final and most heavily filtered phase of the original Apertus data, with no new sources added, packed into 4,096-token sequences and trained on 64 GH200 GPUs at the Swiss National Supercomputing Centre. The cost table puts the 4B run at 2.0E22 FLOPs and the single pass that produced the teacher’s logits for all three students at 1.4E22, against 3.7E23 FLOPs for the original Apertus 8B pre-training on 15T tokens, which the authors describe as making the whole family cost under 12% of the teacher. Swiss AI released 24 checkpoints in total, base, instruction-tuned and quantised, under the Apache 2.0 licence with an acceptable use policy attached, along with the training, post-training and quantisation code. Source: https://arxiv.org/abs/2605.29128 Read the article: https://llmobile.kavents.com/ticker/apertus-mini/Swiss AIDistillationQuantisationOpen weightsApple Siliconhttps://llmobile.kavents.com/ticker/small-vlms-video-compressors/https://llmobile.kavents.com/ticker/small-vlms-video-compressors/Thu, 09 Apr 2026 13:40:25 +0200Researchers at Meta AI and KAUST published Tempo on April 9, 2026, a way to hand a long video to a small vision-language model first and let a larger language model answer the question afterwards. On LVBench, a benchmark whose clips average 4101 seconds, the authors report their 6B system scoring 52.7 against 48.2 for VideoChat-Flash 7B, the strongest specialised baseline in their table, and against 30.8 for GPT-4o and 33.1 for Gemini 1.5 Pro. The small model does the reading and the large one does the reasoning. A Qwen3-VL-2B-Instruct compressor takes an eight-frame chunk of video together with the user’s question and writes out a short run of memory tokens that hold what the chunk contains, and a Qwen3 4B language model then reads the whole condensed sequence with timestamps attached and produces the answer. A linear projector joins the two, for 6B parameters in total. The compressor is the part that could in principle sit on a phone, since it never sees more than a few seconds of video at a time. How many tokens each chunk gets is decided during the same forward pass. The authors insert a yes or no question about whether the segment is relevant to the query, read the difference between the two logits as a score, and keep more memory tokens for chunks that score high, spending between 0.5 and 16 tokens per frame under a fixed overall budget of 4K or 8K visual tokens. On LVBench under the 4K budget the measured average comes to 2.9 tokens per frame. The tighter budget also scores better than the looser one on that benchmark, 52.7 against 52.3, and on the long subset of Video-MME, 57.8 against 57.0. Raising the cap to 2048 frames and a 12K budget brings LVBench to 53.7. Diagram: Fei et al., Figure 2 of the paper, licensed CC BY 4.0. On the other three benchmarks the authors report 67.8 on Video-MME under the 4K budget against 65.3 for VideoChat-Flash and 61.9 for the Qwen3-VL 2B model the compressor starts from, plus 75.6 on MLVU and 65.1 on LongVideoBench. All of these are accuracy scores run through lmms-eval, not speed or memory figures. Nothing in the paper was measured on a phone, a tablet or any other edge device. Training used 32 and then 64 Nvidia H100 GPUs, the paper reports no latency, throughput or memory footprint for the compressor on mobile hardware, and the on-device case rests on the size of the model rather than on a measurement. The authors state that all training datasets they used are publicly available, and they link code and a project page from the paper, with the repository carrying an Apache 2.0 license. They describe no weights release in the paper itself. Their stated next steps are to let the compressor decide for itself when it has gathered enough evidence to stop, which they say would bottleneck inference latency as things stand. Source: https://arxiv.org/abs/2604.08120 Read the article: https://llmobile.kavents.com/ticker/small-vlms-video-compressors/MetaQwenResearchBenchmarksDistillationhttps://llmobile.kavents.com/ticker/mobilellm-pro/https://llmobile.kavents.com/ticker/mobilellm-pro/Mon, 10 Nov 2025 06:28:00 +0100Meta Reality Labs published the MobileLLM-Pro technical report on November 10, 2025, describing a 1.08B-parameter language model built for phones and wearables. The model handles a context window of 128,000 tokens, four times what Gemma 3 1B and Qwen 3 0.6B accept. Quantising weights and embeddings to 4 bits takes the file from 2.2 GB in bf16 down to 590 MB for the CPU build and 720 MB for the accelerator build, which cannot share embedding weights. Meta exported the model through ExecuTorch and measured it on a Samsung Galaxy S25 CPU with the XNNPACK backend and on a Samsung Galaxy S24 Hexagon Tensor Processor. On a 2,000-token prompt the paper reports 8.9 s of prefill on the CPU against 2.0 s on the Hexagon unit, and decode speeds of 33.6 tok/s and 31.6 tok/s while generating 1,000 tokens. At 8,000 tokens prefill climbs to 63.5 s on the CPU and 9.8 s on the Hexagon unit, decode falls to 19.7 and 22.8 tok/s, and the key-value cache grows from 14 MB to 40 MB. The architecture puts 30 layers of width 1280 behind 20 attention heads and 4 key-value heads, and shares one weight matrix between the input embedding and the output head, which the authors say saves 260M parameters, about a quarter of the model. Three out of every four layers use local attention, comparing each token only against the 512 most recent ones, with a global layer every fourth one and at both ends. Meta’s model card credits that interleaving with cutting the key-value cache at 8,000 tokens from 117 MB to 40 MB and with 1.8 times lower prefill latency than global attention in every layer. Meta trained the model by distilling the logits of Llama 4 Scout under a KL divergence loss across three pretraining phases on roughly 1.64T tokens of open data, the second of them a step the authors call implicit positional distillation, which stretches the context window to 128k by having the teacher pass on the ability rather than feeding the student long documents. Meta puts the base model 5.7 and 7.9 points above Gemma 3 1B and Llama 3.2 1B on average across the benchmarks its model card lists, and the paper’s table has it at 76.24 on BoolQ against 63.20 and 62.51, at 52.62 on ARC-Challenge against 38.40 and 38.28, and at 100% needle-in-a-haystack retrieval against 96.80 for Llama 3.2 1B. The instruction-tuned version scores 59.8 on HumanEval against 41.5 and 37.8, 44.8 on MMLU against Gemma’s 29.9 and Llama’s 49.3, and 62.0 on IFEval against Gemma’s 80.2. In Meta’s own human evaluation over 100 prompts per task, raters preferred Gemma 3 1B for summarisation 51 to 47 and for rewriting 49 to 45, while MobileLLM-Pro led on recall 47 to 30. Meta released four checkpoints on Hugging Face, the base model, the instruction-tuned one, and two quantisation-ready base checkpoints produced by quantisation-aware training with self-distillation from the full-precision model. One uses 4-bit group-wise weights at group size 32 with 8-bit dynamic activations and an 8-bit key-value cache for CPU backends, the other 4-bit channel-wise weights for the Apple Neural Engine and Qualcomm’s Hexagon processor, which handle group-wise quantisation poorly or not at all. Needle-in-a-haystack retrieval holds at 96.44% and 98.67% against 100% at full precision. The weights sit behind a request form and carry the FAIR Noncommercial Research License. Source: https://arxiv.org/abs/2511.06719 Read the article: https://llmobile.kavents.com/ticker/mobilellm-pro/MetaResearchDistillationQuantisationAndroidhttps://llmobile.kavents.com/ticker/mobilellm-r1/https://llmobile.kavents.com/ticker/mobilellm-r1/Fri, 12 Sep 2025 12:00:00 +0200Meta released MobileLLM-R1 on September 12, 2025, a family of reasoning models at 140M, 360M and 950M parameters that write out a chain of intermediate steps before giving an answer instead of replying directly. Meta reports the 950M model scoring 74.0 on the MATH500 maths benchmark and 19.9 on LiveCodeBench v6, against 73.0 and 14.9 for Qwen3-0.6B. The technical report followed on September 29, 2025. The three models saw 4.2T training tokens, which Meta puts at 11.7% of the 36T tokens behind Qwen3-0.6B. Pre-training ran in two phases of 2T tokens each, resampled from a curated pool of roughly 2T tokens of open datasets, followed by two mid-training phases of 100B tokens that distil from Llama-3.1-8B-Instruct as a teacher, then supervised fine-tuning on reasoning traces. Meta uses the result to argue that reasoning does not require the corpora above 10T tokens that recent models have been trained on. For the post-trained models Meta’s table puts MobileLLM-R1-950M at 74.0 on MATH500, 67.5 on GSM8K, 15.5 on AIME 2024 and 19.9 on LiveCodeBench v6, where Qwen3-0.6B reaches 73.0, 79.2, 11.3 and 14.9 and DeepSeek-R1-Distill-Qwen-1.5B reaches 83.2, 77.3, 29.1 and 19.9. Among the models whose weights, data and recipe are all public, the same AIME figure compares with 0.6 for OLMo-2 1.48B and 0.3 for SmolLM2-1.7B. On code the 950M base model scores 46.3 on HumanEval against 30.5 for Qwen3-0.6B-Base, and the 140M base model reaches 15.9 where SmolLM2-135M scores 0.0. Meta profiled the models through ExecuTorch on a Samsung Galaxy S22 with 8 GB of RAM, quantised to 4-bit weights with 8-bit dynamic activations. The 140M model generated 129.67 tok/s at a 1k context and 79.71 tok/s at 32k, the 950M model 31.05 tok/s at 1k before running out of memory at 16k, and Llama-3.2-1B 28.71 tok/s at 1k with the same memory failure. The architecture follows Meta’s earlier MobileLLM work and Llama 3.2, sharing one weight matrix between the input embedding and the output layer, and the post-trained models take a 32k context. Meta released the weights on Hugging Face together with the training code, the data sources and the data mixing ratios on GitHub, under the FAIR Noncommercial Research License, which covers research, education and analysis but not commercial use. Access to the model repositories is gated behind a form. Update, November 24, 2025. Meta released MobileLLM-R1.5 at the same three sizes, fine-tuned from the R1 models with one further round of on-policy distillation, where the student answers first and the teacher, here Llama-3.1-Nemotron-Nano-4B-v1.1, grades its own words back to it token by token. Meta reports the 950M version at 86.6 on MATH500, 82.6 on GSM8K, 39.9 on AIME 2024 and 29.1 on LiveCodeBench v6, up from 74.0, 67.5, 15.5 and 19.9 for MobileLLM-R1-950M, and says it beats DeepSeek-R1-Distill-Qwen-1.5B on every maths and coding benchmark it ran. At 360M Meta puts MATH500 at 63.4 against 28.4 and GSM8K at 52.8 against 24.5. Source: https://arxiv.org/abs/2509.24945 Read the article: https://llmobile.kavents.com/ticker/mobilellm-r1/MetaResearchBenchmarksDistillationAndroidhttps://llmobile.kavents.com/ticker/apple-foundation-models-report-2025/https://llmobile.kavents.com/ticker/apple-foundation-models-report-2025/Fri, 18 Jul 2025 01:37:00 +0200Apple posted its 2025 foundation models tech report to arXiv on July 17, 2025, giving the numbers behind the roughly 3-billion-parameter on-device model it shipped at WWDC25. The report’s Table 3 puts the price of compressing that model to 2 bits per weight at 3.4 points of MMLU, from 67.8 in 16 bits down to 64.4, with IFEval falling from 85.1 to 82.3. Apple reaches 2 bits through quantisation-aware training with a learnable scaling factor per weight tensor, and reports that a balanced set of four levels at -1.5, -0.5, 0.5 and 1.5 trains more smoothly than the usual -2, -1, 0 and 1. The embedding table is quantised to 4 bits and the key-value cache to 8 bits, and low-rank adapters are fine-tuned afterwards to claw back the quality the compression costs. The server model takes a different route, compressed after training to 3.56 bits per weight with Adaptive Scalable Texture Compression, a GPU texture format whose fixed-function decode hardware on Apple GPUs unpacks the weights at no cost to the compute cores. The report also spells out the KV cache sharing the framework announcement only named. Apple splits the transformer so that Block 1 holds 62.5 percent of the layers and Block 2 the remaining 37.5 percent with its key and value projections removed, reusing Block 1’s cache instead. That cuts KV cache memory by 37.5 percent, and because Block 2 produces no keys or values, prefill skips its computation entirely, which Apple says cuts time to first token by about the same 37.5 percent. Apple rebuilt how the on-device model is trained. The company first trained a dense model on about 14T tokens, sparse-upcycled it into a 64-expert mixture-of-experts version, where only part of the model runs per token, using 1T high-quality tokens, then retrained the dense model on its last 10 percent of tokens against a distillation loss from that MoE teacher. Apple states this cut the cost of training the teacher by 90 percent and removed the need for structural pruning. The server model was trained on 8192 v5p Cloud TPU chips for 13.4T tokens. For the language expansion the report states the models are designed to support 16 languages, built on a text tokenizer grown from a 100k to a 150k vocabulary and a continued pre-training mix whose multilingual share rose from 8 percent to 30 percent. In Apple’s own human evaluation on US English text prompts, graders preferred the on-device model over Qwen-2.5-3B on 35.3 percent of prompts against 12.9 percent the other way, while against Gemma-3-4B the split was 21.0 percent to 21.9 percent. The report is on arXiv under a Creative Commons Attribution 4.0 license. Chart: Apple's own human evaluation, from the tech report. Rows marked with an asterisk were tested against the compressed on-device model. Source: https://arxiv.org/abs/2507.13575 Read the article: https://llmobile.kavents.com/ticker/apple-foundation-models-report-2025/AppleApple SiliconQuantisationDistillationBenchmarkshttps://llmobile.kavents.com/ticker/slms-can-still-pack-a-punch/https://llmobile.kavents.com/ticker/slms-can-still-pack-a-punch/Fri, 03 Jan 2025 20:53:57 +0100Shreyas Subramanian, Vikram Elango and Mecit Gungor of Amazon surveyed about 160 papers on small language models on January 3, 2025, covering general-purpose models under 8B parameters plus a few exceptions up to 13B. Their central claim is that some of these models reach the performance of models 10 to 100 times their parameter count, a ratio the authors call effective size and anchor to PaLM at 540B parameters. In their table, Phi-1 at 1.3B parameters is credited with an effective size of 175B on code benchmarks and Phi-2 at 2.7B with 70B. The survey is clear about where that stops. Its appendix prints completions from 14 models for the prompt “The diagonals of a rectangle”, where a 33M TinyStories model answers “and they were so beautiful” and a 125M GPT Neo model loops on its own words, while TinyLlama at 1.1B and Phi 1.5 at 1.3B give the correct definition. The authors conclude that several billion parameters are the minimum for usable general-purpose responses, and they caution that every effective size they report comes from each paper’s best reported benchmark, so results on other benchmarks can be much lower. The clearest wins they list are task-specific and stay inside one domain, with WizardMath at 7B beating Llama 2 70B on maths benchmarks and Code Llama at 7B beating it on coding benchmarks. On what closes the gap, the survey puts training data first. It credits Phi-1 with 50.6 percent on HumanEval at 1.3B parameters trained on textbook-quality synthetic data, and TinyGSM, where a 1.3B generator paired with a 1.3B verifier scored 81.5 on the GSM8K maths set against 77.4 for the GPT-3.5 model that wrote its training data. Distillation covers the training side, with Orca 2 at 13B leading Llama 2 70B by an average of 3.23 points after learning step-by-step explanations from a teacher model rather than just its answers. After training, the survey groups quantisation and pruning, citing SmoothQuant for 8-bit weights and activations at up to 1.56 times the speed and half the memory, and Sheared-LLaMA for cutting Llama 2 7B down to 1.3B and 2.7B using 3 percent of the compute that training those sizes from scratch would need. The authors name resource-constrained deployment as the main motivation for the whole field, listing mobile devices, edge computing and regions with limited connectivity, and point at SmolLM at 135M, 360M and 1.7B parameters as a family built to run locally on phones and laptops. They note that extreme quantisation and sparse training already put Llama inference on CPUs within reach, and that 8-bit Adam training of a 7B model fits on a single 24 GB Nvidia RTX 4090. The paper is on arXiv under a CC BY-NC-ND 4.0 license, and its authors maintain a leaderboard of small models on Hugging Face Spaces. Source: https://arxiv.org/abs/2501.05465 Read the article: https://llmobile.kavents.com/ticker/slms-can-still-pack-a-punch/ResearchBenchmarksDistillationQuantisationPhihttps://llmobile.kavents.com/ticker/gemma-2/https://llmobile.kavents.com/ticker/gemma-2/Wed, 31 Jul 2024 17:00:00 +0200Google released Gemma 2 2B on July 31, 2024, an open model of 2.6B parameters that Google says produces outsized results by learning from larger models through distillation. Google reports it at 1126 Elo on the LMSYS Chatbot Arena leaderboard, captured on July 30, 2024, ahead of Mixtral 8x7B Instruct at 1114, GPT 3.5 Turbo 0314 at 1106 and Llama 2 70B chat at 1093, and states that the model surpasses all GPT-3.5 models there. Chart: Google, from the announcement. Scores are Google's own reading of the leaderboard on July 30, 2024. The technical report describes training the 2B size on 2 trillion tokens with knowledge distillation, where a bigger teacher model supplies a probability for every token in context and the small student is trained to match those probabilities rather than to predict the next token. In an ablation the authors ran, a 2B model trained over 500B tokens distilled from a 7B teacher averaged 67.7 across three benchmarks, against 60.3 for the same model trained from scratch. The released model holds 2.02B non-embedding parameters plus 590M embedding parameters, which come from the 256k-entry Gemini vocabulary, and it interleaves local sliding-window attention with global attention over an 8192-token context. Google DeepMind trained it on 512 TPUv5e chips. Google names edge devices and laptops as deployment targets next to cloud serving on Vertex AI and Google Kubernetes Engine, and says the model is small enough for the free tier of T4 GPUs in Google Colab. For running it locally Google points to Ollama and to Gemma.cpp, its own lightweight C++ inference engine, and says support in MediaPipe, the on-device inference stack for Android, iOS and the web, is coming. Google adds that NVIDIA optimised the model with the TensorRT-LLM library for GeForce RTX cards and for Jetson modules at the edge. Google put the weights on Kaggle, Hugging Face and Vertex AI Model Garden under the Gemma terms, which it calls commercially friendly for research and commercial applications, and made the model available to try in Google AI Studio. The same announcement added ShieldGemma, a set of safety classifiers built on Gemma 2 that filter hate speech, harassment, sexually explicit and dangerous content, and Gemma Scope, over 400 sparse autoencoders for inspecting the inner workings of Gemma 2 2B and 9B. Source: https://developers.googleblog.com/en/smaller-safer-more-transparent-advancing-responsible-ai-with-gemma/ Read the article: https://llmobile.kavents.com/ticker/gemma-2/GoogleGemmaDistillationOpen weightsBenchmarkshttps://llmobile.kavents.com/ticker/lamini-lm/https://llmobile.kavents.com/ticker/lamini-lm/Thu, 27 Apr 2023 19:58:00 +0200Researchers at MBZUAI, the University of British Columbia and Monash University published LaMini-LM on April 27, 2023, a family of 17 distilled instruction-following models ranging from 61M to 7B parameters. The paper gives resource use as the motivation, stating that earlier distilled models of 7B to 13B parameters present “challenges for deployment in resource-constrained settings” and that the authors set out to make deployment in such settings easier. The paper names no target device and reports no memory footprint, no quantisation and no generation speed, and the authors state they trained and evaluated on V100 and A100 GPUs. The models are fine-tuned on the LaMini instruction dataset, 2.58M instruction and response pairs that the team generated with gpt-3.5-turbo from prompts taken from self-instruct, P3, FLAN and Alpaca. The authors describe it as the largest instruction dataset of its kind at the time and 50 times the size of the dataset behind Alpaca. The students are 6 encoder-decoder models built on T5 and Flan-T5 and 11 decoder-only models built on GPT-2, GPT-Neo, Cerebras-GPT, GPT-J and LLaMA. The authors evaluated the family on 15 NLP benchmarks with EleutherAI’s lm-evaluation-harness and report that LaMini-Flan-T5-248M performs on par with LLaMA-7B on the benchmark average, against a baseline with about 28 times as many parameters. They warn in the repository that their LLaMA figures are not comparable to the ones the LLaMA authors published, because they re-ran the evaluation themselves. A human rating of 114 user-oriented instructions follows the same size trend and leaves a gap to gpt-3.5-turbo, and the paper reports that the models do poorly on coding, maths and tasks that need reasoning. MBZUAI publishes the weights and the instruction dataset on Hugging Face. The repository puts the code under Apache 2.0 and the models and the data under CC BY-NC 4.0, and states that both are intended for research use only. The paper was later accepted at the EACL 2024 main conference. Source: https://arxiv.org/abs/2304.14402 Read the article: https://llmobile.kavents.com/ticker/lamini-lm/ResearchDistillationOpen weightsBenchmarkshttps://llmobile.kavents.com/ticker/distilbert/https://llmobile.kavents.com/ticker/distilbert/Wed, 02 Oct 2019 19:56:00 +0200Hugging Face published DistilBERT on October 2, 2019, a compressed version of BERT trained by knowledge distillation, where a small student model learns to reproduce the output of a larger teacher. The student carries 66M parameters against BERT-base’s 110M, and Victor Sanh and his co-authors report it retaining 97% of BERT’s score on the GLUE language understanding benchmark while running 60% faster. The authors tested the model on a phone by building a question-answering app for an iPhone 7 Plus and comparing it against their own BERT-base question-answering model. Excluding the tokenisation step, they measured DistilBERT at 71% faster, and the whole model in that app weighs 207 MB, which they note could shrink further with quantisation. On a server CPU at batch size 1, the same pair takes 410 seconds against 668 for a full pass over the STS-B development set. Three losses train the student at once. A distillation loss pushes it towards the teacher’s full output distribution rather than only the correct token, a masked language modelling loss keeps BERT’s own pre-training objective in place, and a cosine embedding loss aligns the directions of student and teacher hidden state vectors. The student halves BERT’s layer count, drops the token-type embeddings and the pooler, and starts from every second layer of the teacher, an initialisation the ablation study values at 3.69 points of GLUE macro score. Training ran on eight 16 GB V100 GPUs for about 90 hours on English Wikipedia and the Toronto Book Corpus. Hugging Face released the trained weights together with the training code in its transformers library, and published the iOS demo separately as Core ML code. The paper was accepted at the fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing, co-located with NeurIPS 2019. Source: https://arxiv.org/abs/1910.01108 Read the article: https://llmobile.kavents.com/ticker/distilbert/ResearchDistillationiPhoneOpen weightshttps://llmobile.kavents.com/ticker/tinybert/https://llmobile.kavents.com/ticker/tinybert/Mon, 23 Sep 2019 15:05:00 +0200Researchers at Huawei Noah’s Ark Lab and Huazhong University of Science and Technology published TinyBERT on September 23, 2019, a compressed version of BERT-base trained by knowledge distillation, where a small student model learns to reproduce the behaviour of a larger teacher. The four-layer student carries 14.5M parameters against the teacher’s 109M, and the authors report it at 7.5x smaller and 9.4x faster on inference while reaching more than 96.8% of the teacher’s score on the GLUE language understanding benchmark. A forward pass costs 1.2B FLOPs against 22.5B for BERT-base. Rather than fitting only the logits the teacher outputs, the authors define losses on the embedding layer output and, for every Transformer layer, on the hidden states and on the attention matrices those layers produce, following a finding by Kevin Clark and co-authors that BERT’s attention weights capture substantial linguistic knowledge. Dropping the attention term costs the most in their ablation, taking a four-task development average from 75.6 to 71.0, against 72.9 without the hidden states and 73.5 without the prediction layer logits. The distillation also runs twice, first as a general stage against the unfine-tuned BERT on a large text corpus, which produces a general TinyBERT that initialises the student, then as a task-specific stage against the fine-tuned BERT on a task dataset expanded by data augmentation. Removing either stage costs more still, with that same average falling to 72.5 without the general stage and 68.5 without the task-specific one. On the GLUE test set the authors report the 14.5M-parameter model at an average of 77.0 against 79.5 for BERT-base, with 82.5 on MNLI-m, 92.6 on SST-2, 87.7 on QNLI, 71.3 on QQP, 86.4 on MRPC, 80.4 on STS-B and 66.6 on RTE. The widest gap is CoLA, the linguistic acceptability task, at 44.1 against 52.8. The authors measured the speedups on a single NVIDIA K80 GPU, not on a phone. A six-layer variant at 67M parameters averages 79.4, which the authors call on par with the teacher. Huawei released the checkpoints on Hugging Face, where the general four-layer model is published as huawei-noah/TinyBERT_General_4L_312D alongside a six-layer version, with the training code in the company’s Pretrained-Language-Model repository on GitHub. The paper later appeared in Findings of EMNLP 2020. Source: https://arxiv.org/abs/1909.10351 Read the article: https://llmobile.kavents.com/ticker/tinybert/ResearchDistillationOpen weightsBenchmarks