https://llmobile.kavents.com/tags/phi/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/confidant-collaborative-training/https://llmobile.kavents.com/ticker/confidant-collaborative-training/Mon, 03 Nov 2025 10:00:00 +0100Researchers at Zhejiang University, Zhejiang University of Technology and Hangzhou Dianzi University published Confidant in the ACM MobiCom 2025 proceedings on November 3, 2025, a day before the conference opened in Hong Kong. The framework fine-tunes a language model across several off-the-shelf devices instead of one, and the authors fine-tuned Phi2-2.7B on the Alpaca instruction dataset in 40.1 hours using three of them. Their write-up of that run names a Redmi K50 phone, a Core i7-13650HX laptop and a MacBook Pro with an M1 Pro, while the paper’s abstract and conclusion describe the same result as three mobile devices. The same trio fine-tuned BERT-Base in 2.2 hours. Confidant cuts a model into consecutive slices, gives each device one slice, and passes activations from one device to the next, so no device ever holds the whole model. Batches flow through that chain as a pipeline, with each device starting a new batch while the devices behind it are still working on earlier ones. The device that holds the training data acts as coordinator and recalculates where the cuts fall during training, using each participant’s measured compute time, free memory and bandwidth, and it discards any split whose slice would not fit in a device’s memory. Inside a device, a second scheduler profiles the CPU and the GPU on varying numbers of attention heads and gives each processor as many heads as it can finish in roughly the same time, which is what the authors credit for running up to 1.94 times faster than a capacity-aware pipeline baseline they implemented themselves, and up to 3.82 times faster than the best single device, both on GPT-2 Medium. Diagram: Chen et al., Figure 3 of the paper, licensed CC BY 4.0. Splitting the model is what brings the footprint per device down. The paper’s memory table puts fine-tuning BERT-Base at a batch size of 8 at 5.5 GB on a single Redmi K50, against a per-device average of 3.17 GB across three devices and 2.43 GB across four. Larger models leave no single-device figure to compare with, because a Redmi K50 fits only 1 of Phi2-2.7B’s 32 transformer blocks at a batch size of 4, and the per-device averages of 15.31 GB for Phi2-2.7B and 32.07 GB for LLaMA3-8B across three devices are carried by the two laptops rather than the phone. A separate table puts the memory PyTorch needs to fine-tune Phi2-2.7B at a batch size of 8 at 46.5 GB, next to the 8 GB to 12 GB of the four phones it lists. The paper’s prose says Confidant almost does not cause any accuracy drop, while its own model performance table puts the three-device runs at a test loss of 0.66 on Phi2-2.7B and 3.55 on LLaMA3-8B, against 0.52 and 2.86 for fine-tuning the same models the conventional way on a single machine, where a lower loss is better. The four-device runs land at 0.76 and 2.85 on that same comparison, and BERT-Base token classification accuracy holds at 97.66% against 97.70%. Keeping data on the device is the stated motivation. The authors write that training with Confidant does not require sharing raw data and that devices exchange only intermediate outputs and model gradients, which they present as the privacy argument for their two scenarios, a household assistant fine-tuned on family data during idle hours and a conference assistant fine-tuned on attendees’ phones. They also state that fine-tuning on a single cloud server would be faster and more straightforward, and that their aim is to use idle mobile hardware instead. The Android side is built on Alibaba’s MNN framework with laptops running PyTorch, about 10,900 lines of code in total, the paper is published under a Creative Commons Attribution 4.0 license, and the authors wrote that they planned to release the code in late 2025. Source: https://dl.acm.org/doi/10.1145/3680207.3723487 Read the article: https://llmobile.kavents.com/ticker/confidant-collaborative-training/ResearchMemoryPhiLlamaAndroidhttps://llmobile.kavents.com/ticker/slms-can-still-pack-a-punch/https://llmobile.kavents.com/ticker/slms-can-still-pack-a-punch/Fri, 03 Jan 2025 20:53:57 +0100Shreyas Subramanian, Vikram Elango and Mecit Gungor of Amazon surveyed about 160 papers on small language models on January 3, 2025, covering general-purpose models under 8B parameters plus a few exceptions up to 13B. Their central claim is that some of these models reach the performance of models 10 to 100 times their parameter count, a ratio the authors call effective size and anchor to PaLM at 540B parameters. In their table, Phi-1 at 1.3B parameters is credited with an effective size of 175B on code benchmarks and Phi-2 at 2.7B with 70B. The survey is clear about where that stops. Its appendix prints completions from 14 models for the prompt “The diagonals of a rectangle”, where a 33M TinyStories model answers “and they were so beautiful” and a 125M GPT Neo model loops on its own words, while TinyLlama at 1.1B and Phi 1.5 at 1.3B give the correct definition. The authors conclude that several billion parameters are the minimum for usable general-purpose responses, and they caution that every effective size they report comes from each paper’s best reported benchmark, so results on other benchmarks can be much lower. The clearest wins they list are task-specific and stay inside one domain, with WizardMath at 7B beating Llama 2 70B on maths benchmarks and Code Llama at 7B beating it on coding benchmarks. On what closes the gap, the survey puts training data first. It credits Phi-1 with 50.6 percent on HumanEval at 1.3B parameters trained on textbook-quality synthetic data, and TinyGSM, where a 1.3B generator paired with a 1.3B verifier scored 81.5 on the GSM8K maths set against 77.4 for the GPT-3.5 model that wrote its training data. Distillation covers the training side, with Orca 2 at 13B leading Llama 2 70B by an average of 3.23 points after learning step-by-step explanations from a teacher model rather than just its answers. After training, the survey groups quantisation and pruning, citing SmoothQuant for 8-bit weights and activations at up to 1.56 times the speed and half the memory, and Sheared-LLaMA for cutting Llama 2 7B down to 1.3B and 2.7B using 3 percent of the compute that training those sizes from scratch would need. The authors name resource-constrained deployment as the main motivation for the whole field, listing mobile devices, edge computing and regions with limited connectivity, and point at SmolLM at 135M, 360M and 1.7B parameters as a family built to run locally on phones and laptops. They note that extreme quantisation and sparse training already put Llama inference on CPUs within reach, and that 8-bit Adam training of a 7B model fits on a single 24 GB Nvidia RTX 4090. The paper is on arXiv under a CC BY-NC-ND 4.0 license, and its authors maintain a leaderboard of small models on Hugging Face Spaces. Source: https://arxiv.org/abs/2501.05465 Read the article: https://llmobile.kavents.com/ticker/slms-can-still-pack-a-punch/ResearchBenchmarksDistillationQuantisationPhihttps://llmobile.kavents.com/ticker/phi-3-mini/https://llmobile.kavents.com/ticker/phi-3-mini/Mon, 22 Apr 2024 18:00:00 +0200Microsoft introduced Phi-3-mini on April 22, 2024, a 3.8-billion-parameter model trained on 3.3 trillion tokens. The technical report puts it at 69 percent on MMLU and 8.38 on MT-bench, which the authors say rivals models such as Mixtral 8x7B and GPT-3.5. Quantised to 4 bits the model occupies about 1.8 GB, small enough for a phone. The report describes the test directly. We tested the quantized model by deploying phi-3-mini on iPhone 14 with A16 Bionic chip running natively on-device and fully offline achieving more than 12 tokens per second. Screenshots: Microsoft, Figure 1 of the technical report. The same report covers two larger siblings, Phi-3-small at 7B parameters and Phi-3-medium at 14B, both trained on 4.8 trillion tokens. Microsoft reports 75 percent on MMLU for the small model and 78 percent for the medium one. Source: https://arxiv.org/abs/2404.14219 Read the article: https://llmobile.kavents.com/ticker/phi-3-mini/MicrosoftPhiiPhoneQuantisationOpen weightshttps://llmobile.kavents.com/ticker/tinyllava/https://llmobile.kavents.com/ticker/tinyllava/Thu, 22 Feb 2024 06:05:30 +0100Researchers at Beihang University and Tsinghua University posted TinyLLaVA on February 22, 2024, a framework for building small vision-language models out of an existing language model and an existing image encoder. Their best configuration, which the paper calls TinyLLaVA-3.1B, scores higher than the 7B LLaVA-1.5 on seven of the nine benchmarks in the paper’s comparison table, ties it on GQA at 62.0, and trails it on MME, where the same table gives LLaVA-1.5 1510.7 against 1464.9. The authors report 79.9 against 78.5 on VQAv2 and 75.8 against 63.4 on LLaVA-Bench-in-the-Wild, and put the model ahead of the 7B Qwen-VL on four of the five benchmarks they share, behind it on TextVQA at 59.1 against 63.8. The framework treats four things as variables, the small language model, the vision encoder that turns an image into patch features, the connector that maps those features into the language model’s embedding space, and the training recipe. The authors instantiate it with TinyLlama at 1.1B, StableLM-2 at 1.6B and Phi-2 at 2.7B as language models, CLIP at 0.3B and SigLIP at 0.4B as vision encoders, and either the 558k-caption LLaVA-1.5 data or the 1246k-caption ShareGPT4V data. Their base recipe updates only the connector during pre-training, while their share recipe freezes the first 12 layers of the vision encoder and updates the rest of the model. The ablations put most of the movement in the vision encoder and the recipe. Swapping CLIP for SigLIP raises the Phi-2 variant’s TextVQA score from 51.4 to 55.6 and the TinyLlama variant’s from 45.8 to 49.1, which the authors attribute in part to SigLIP running at 384 pixels against 336 and emitting 729 visual tokens against 576. A two-layer MLP connector beat a resampler on four of six benchmarks in the single comparison the paper runs, 74.0 against 70.1 on VQAv2 and 58.1 against 54.2 on GQA, with the resampler ahead on SQA-I and MM-Vet. The authors report that the share recipe, which fine-tunes part of the vision encoder rather than freezing all of it, improved every variant pre-trained on ShareGPT4V. Model size still helps when data and recipe are held fixed, and with CLIP under the base recipe the Phi-2 variant leads the StableLM-2 and TinyLlama ones on all seven benchmarks in the appendix table. The ordering flips once the recipe changes, with the 1.5B SigLIP TinyLlama model on the share recipe beating the 1.9B CLIP StableLM-2 model on the base recipe on six of seven benchmarks. The authors add a hallucination caveat, reporting that the share recipe lowers POPE accuracy for the StableLM-2 and Phi-2 variants even as their other scores rise, and they conjecture that smaller language models need more trainable parameters during pre-training while larger ones suffer from them. The paper carries no on-device measurement. It states no latency, no memory footprint and no target hardware, and argues the small-scale case from training and inference cost rather than from phone or edge deployment. The authors wrote that weights and code would be made public, and the project repository and the 3.1B weights carry the Apache 2.0 license. Source: https://arxiv.org/abs/2402.14289 Read the article: https://llmobile.kavents.com/ticker/tinyllava/ResearchBenchmarksPhiOpen weightsOpen sourcehttps://llmobile.kavents.com/ticker/phi-2/https://llmobile.kavents.com/ticker/phi-2/Tue, 12 Dec 2023 18:00:00 +0100Microsoft released Phi-2 on December 12, 2023, a base language model with 2.7 billion parameters. The company says that on complex benchmarks it matches or outperforms models up to 25 times larger, and that it surpasses Mistral and Llama-2 at 7B and 13B parameters on various aggregated benchmarks. Microsoft’s own table scores Phi-2 at 53.7 on grouped coding benchmarks against 38.3 for Llama-2-70B, and at 61.1 on math against 64.1 for the same model. Against Google’s Gemini Nano 2 at 3.2B, Microsoft reports 59.3 versus 42.4 on BigBench-Hard and 59.1 versus 27.2 on MBPP, using Google’s own reported figures for the comparison. Microsoft trained the model on 1.4 trillion tokens from multiple passes over a mixture of synthetic and web data for language and code, following what it calls textbook-quality data. The synthetic sets were written to teach common sense reasoning and general knowledge such as science, daily activities and theory of mind, and the web portion was filtered for educational value and content quality. The model card puts the underlying dataset at 250 billion tokens, combining synthetic text generated with GPT-3.5 and web data from Falcon RefinedWeb and SlimPajama that was assessed with GPT-4. Microsoft did not train Phi-2 from scratch. It started from the 1.3-billion-parameter Phi-1.5 and embedded that model’s knowledge inside the larger one, a step the company says both accelerates training convergence and raises Phi-2’s benchmark scores. Training then took 14 days on 96 A100 GPUs, and the model card lists a context length of 2048 tokens. Phi-2 is a base model that went through neither reinforcement learning from human feedback nor instruction tuning. Chart: Microsoft's own evaluation figures for the two models. Microsoft published the weights in the Azure AI Studio model catalog and on Hugging Face, where the first model card, dated December 13, 2023, named the Microsoft Research License and limited use to research. Microsoft replaced that file on January 5, 2024, and the model card has since stated “The model is licensed under the MIT license”, which the repository’s commit history dates to that day. Source: https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ Read the article: https://llmobile.kavents.com/ticker/phi-2/MicrosoftPhiBenchmarksResearchOpen weightshttps://llmobile.kavents.com/ticker/phi-1-5/https://llmobile.kavents.com/ticker/phi-1-5/Mon, 11 Sep 2023 16:00:00 +0200Microsoft Research published phi-1.5 on September 11, 2023, a 1.3-billion-parameter model that applies the synthetic textbook data approach of phi-1 to common sense reasoning and general world knowledge rather than Python code. The authors report common sense benchmark results comparable to models ten times the size trained on more than ten times as much data, and describe natural language performance as comparable to models five times larger. phi-1.5 scores 0.734 on WinoGrande against 0.691 for Llama2-7B, 0.444 on ARC-Challenge against 0.434, and 0.526 on SIQA against 0.480, all measured in Microsoft’s own evaluation pipeline. The gap widens on multi-step reasoning, with 40.2 on GSM8K grade-school maths against 14.6 for Llama2-7B, 34.1 on HumanEval against 12.8, and 37.7 on MBPP against 20.8, which the authors note puts its coding scores at or above Llama-65B. MMLU runs the other way, at 0.376 for phi-1.5 and 0.453 for Llama2-7B. Chart: Microsoft, Figure 1 of the technical report. The training set combines 7B tokens from phi-1 with roughly 20B tokens of newly generated textbook-like text, seeded from 20,000 selected topics spanning science, daily activities and theory of mind. That yields a 30B-token dataset whose only non-synthetic part is 6B tokens of filtered code, and training runs over 150B tokens, 80 percent of them from the new synthetic data. Microsoft puts the cost at 1,500 GPU hours on a single A100-80G, against more than 80,000 hours for Llama-7B. The same report covers two variants built to test what web data adds. phi-1.5-web-only trains purely on 95B tokens of filtered web text, 88B of it from the Falcon RefinedWeb dataset and 7B of code from The Stack and Stack Overflow, while phi-1.5-web mixes filtered web data, phi-1’s code and the synthetic text in shares of roughly 40, 20 and 40 percent. phi-1.5-web comes out ahead on reasoning at 44.6 on GSM8K, 41.4 on HumanEval and 43.5 on MBPP. The authors list hallucinations and the potential for toxic and biased generations among the traits phi-1.5 shares with larger models, and write that it “is not immune” despite the absence of web data in its training set. On an 86-prompt probe they wrote themselves, phi-1.5 passed 47 and failed 34, while Llama2-7B and Falcon-7B failed 54 and 50. Microsoft released the raw base model with no instruction fine-tuning and no RLHF, and the weights sit on Hugging Face under the MIT license. Source: https://arxiv.org/abs/2309.05463 Read the article: https://llmobile.kavents.com/ticker/phi-1-5/MicrosoftPhiResearchBenchmarksOpen weightshttps://llmobile.kavents.com/ticker/phi-1/https://llmobile.kavents.com/ticker/phi-1/Tue, 20 Jun 2023 18:14:00 +0200Microsoft Research published phi-1 on June 20, 2023, a Python code model with 1.3B parameters. The paper reports 50.6 percent pass@1 on HumanEval and 55.5 percent on MBPP, pass@1 being the share of problems the model solves with its first generated answer. The same table lists StarCoder at 15.5B parameters and 1 trillion training tokens with 33.6 percent and 52.7 percent, GPT-3.5 at 175B parameters with 47 percent, and CodeGen-Mono at 16.1B parameters and 577B tokens with 29.3 percent. The training data comes to under 7B tokens in three parts. About 6B tokens are Python files pulled from The Stack and StackOverflow by a classifier the authors trained on roughly 100,000 GPT-4 ratings of how instructive a snippet is for someone learning to code. Under 1B tokens are synthetic Python textbooks written by GPT-3.5, and about 180M tokens are synthetic exercises with solutions, used only in the finetuning step that turns phi-1-base into phi-1. Microsoft ran the training on 8 Nvidia A100 GPUs, reaching the base model in under 4 days and spending another 7 hours on the finetuning. That amounts to about 8 passes over the pretraining data and a little over 50B tokens seen, which the paper’s chart puts at 1090 GPU hours for the 1.3B run. Chart: Microsoft, Figure 2.1 of the paper. The GPU hours on the axis are the authors' own figures. The same pipeline at 350M parameters yields phi-1-small, which the authors put at 45 percent on HumanEval and use to argue that parameter count still drives which capabilities appear at all. Microsoft posted phi-1 on Hugging Face in September 2023, where the model card lists the MIT license. Source: https://arxiv.org/abs/2306.11644 Read the article: https://llmobile.kavents.com/ticker/phi-1/MicrosoftPhiResearchBenchmarksOpen weights