https://llmobile.kavents.com/tags/meta/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/mobilemoe/https://llmobile.kavents.com/ticker/mobilemoe/Tue, 26 May 2026 19:58:00 +0200Meta AI published MobileMoE on May 26, 2026, a family of three language models for phones that hold 1.3B, 2.8B and 5.3B parameters but run only 272M, 528M and 922M of them on any given token. Each one is a sparse mixture of experts, meaning the feedforward block in every layer is split into 60 small networks plus one always-on shared network, and a router picks 4 of the 60 for each token while the rest sit idle. Per-token compute therefore follows the active count, and the memory footprint follows the total. That split is what makes the models awkward to host on a phone. The authors quantise all three to 4 bits with quantisation-aware training, which puts the weights at 0.68 GB, 1.48 GB and 2.75 GB, because every expert has to be reachable even though only a few run per token. On a Samsung Galaxy S25 they measure peak resident memory during inference at 1.49 GB for the smallest model at an 8k context against 1.91 GB for the dense MobileLLM-Pro, and at 4.71 GB for the largest. Sizing the whole expert set to fit phone DRAM is the opposite of the route Edge0 takes, which streams expert weights off storage to keep a 35B model inside a couple of gigabytes. The paper reports that peak memory under real prompts runs 1.2 to 2.1 times higher than under repeated dummy tokens for the mixture-of-experts models, while staying flat for the dense baseline, because varied input routes to more experts and pulls more of them into RAM. Meta AI therefore profiled with real code, knowledge and maths prompts and calls dummy-prompt figures a lower bound rather than a measurement. The authors ran the models through ExecuTorch on the Galaxy S25 with a Snapdragon 8 Elite on 4 CPU threads and on an iPhone 16 Pro with an A18 Pro on 2 CPU threads, after writing a fused mixture-of-experts operator because mobile CPU backends ship none. On a 512-token prompt they measure the smallest model decoding at 130.0 tok/s on the Galaxy S25 against 56.5 for MobileLLM-Pro, and at 180.5 tok/s on the iPhone 16 Pro against 55.1. Time to first token on a 1k prompt drops from 4.26 s to 2.01 s on the Galaxy and from 6.03 s to 2.14 s on the iPhone. Across both phones, CPU and GPU, and the XNNPACK and MLX backends, Meta AI reports the smallest model running 1.8 to 3.8 times faster at prefill and 2.2 to 3.4 times faster at decode than the dense baseline at comparable 4-bit weight memory. Across 14 benchmarks the authors score the instruction-tuned models at 46.7, 55.3 and 60.1 on average at 16-bit precision, against 42.7 for the dense Qwen3.5 0.8B, 44.0 for Gemma 3 1B, 46.9 for Llama 3.2 1B and 50.8 for Qwen3.5 2B, the last of which carries 1.9B active parameters. The largest model reaches 60.1 against 55.6 for Ai2’s OLMoE-1B-7B, with 30 percent fewer active and 23 percent fewer total parameters. After 4-bit quantisation those averages fall to 44.0, 52.5 and 57.8, which leaves the smallest model 1.5 points behind MobileLLM-Pro at 45.5 while scoring 9.4 points higher on MMLU. Meta AI states that the training data across all four stages comes from openly licensed public datasets, names no weight release, and lists mobile NPU deployment, expert pruning and mixed-precision quantisation as further work. Chart: Meta AI. The authors' own comparison of instruction-tuned models at 16-bit precision, with the horizontal axis giving total parameters alongside the 4-bit weight footprint those imply. Source: https://arxiv.org/abs/2605.27358 Read the article: https://llmobile.kavents.com/ticker/mobilemoe/MetaMixture of expertsMemoryQuantisationiPhonehttps://llmobile.kavents.com/ticker/small-vlms-video-compressors/https://llmobile.kavents.com/ticker/small-vlms-video-compressors/Thu, 09 Apr 2026 13:40:25 +0200Researchers at Meta AI and KAUST published Tempo on April 9, 2026, a way to hand a long video to a small vision-language model first and let a larger language model answer the question afterwards. On LVBench, a benchmark whose clips average 4101 seconds, the authors report their 6B system scoring 52.7 against 48.2 for VideoChat-Flash 7B, the strongest specialised baseline in their table, and against 30.8 for GPT-4o and 33.1 for Gemini 1.5 Pro. The small model does the reading and the large one does the reasoning. A Qwen3-VL-2B-Instruct compressor takes an eight-frame chunk of video together with the user’s question and writes out a short run of memory tokens that hold what the chunk contains, and a Qwen3 4B language model then reads the whole condensed sequence with timestamps attached and produces the answer. A linear projector joins the two, for 6B parameters in total. The compressor is the part that could in principle sit on a phone, since it never sees more than a few seconds of video at a time. How many tokens each chunk gets is decided during the same forward pass. The authors insert a yes or no question about whether the segment is relevant to the query, read the difference between the two logits as a score, and keep more memory tokens for chunks that score high, spending between 0.5 and 16 tokens per frame under a fixed overall budget of 4K or 8K visual tokens. On LVBench under the 4K budget the measured average comes to 2.9 tokens per frame. The tighter budget also scores better than the looser one on that benchmark, 52.7 against 52.3, and on the long subset of Video-MME, 57.8 against 57.0. Raising the cap to 2048 frames and a 12K budget brings LVBench to 53.7. Diagram: Fei et al., Figure 2 of the paper, licensed CC BY 4.0. On the other three benchmarks the authors report 67.8 on Video-MME under the 4K budget against 65.3 for VideoChat-Flash and 61.9 for the Qwen3-VL 2B model the compressor starts from, plus 75.6 on MLVU and 65.1 on LongVideoBench. All of these are accuracy scores run through lmms-eval, not speed or memory figures. Nothing in the paper was measured on a phone, a tablet or any other edge device. Training used 32 and then 64 Nvidia H100 GPUs, the paper reports no latency, throughput or memory footprint for the compressor on mobile hardware, and the on-device case rests on the size of the model rather than on a measurement. The authors state that all training datasets they used are publicly available, and they link code and a project page from the paper, with the repository carrying an Apache 2.0 license. They describe no weights release in the paper itself. Their stated next steps are to let the compressor decide for itself when it has gathered enough evidence to stop, which they say would bottleneck inference latency as things stand. Source: https://arxiv.org/abs/2604.08120 Read the article: https://llmobile.kavents.com/ticker/small-vlms-video-compressors/MetaQwenResearchBenchmarksDistillationhttps://llmobile.kavents.com/ticker/mobilellm-flash/https://llmobile.kavents.com/ticker/mobilellm-flash/Mon, 16 Mar 2026 23:10:00 +0100Meta AI published MobileLLM-Flash on March 16, 2026, three models of 350M, 650M and 1.4B parameters whose shapes were picked by measuring how fast each candidate actually ran on a phone. Meta reports the family reaching up to 1.8 times faster prefill and 1.6 times faster decode than Liquid AI’s LFM2 models on mobile CPUs, at comparable or higher accuracy, with support for up to 8k tokens of context. Meta’s first MobileLLM paper argued in 2024 that deep and thin networks win below a billion parameters, and the Flash paper states that such deep-and-thin structures often fail to improve on-device latency. Meta exported 100 architectures to a phone and found parameter count a poor predictor of measured prefill and decode speed, at a Kendall rank correlation of 0.40, with FLOPs little better at 0.46 and 0.55. The 1.4B model therefore runs 16 layers at a width of 2048, against the 30 layers at 1280 of MobileLLM-Pro-1B. Meta ran the latency tests on a Samsung Galaxy S25, a phone with a Snapdragon 8 Elite chipset, an octa-core CPU and 12 GB of memory, exporting each model through ExecuTorch to the XNNPACK CPU backend at 4-bit weights, 8-bit dynamic activations and a quantised KV cache, on 4 CPU threads and averaged over three runs after a warmup. At a 2k context the 650M model reaches its first token in 3.34 s against 6.01 s for LFM2 700M, and decodes at 85.35 tok/s against 53.57 tok/s. Meta repeated the 1k-context test on an iPhone 17, where the 650M model takes 2.81 s to the first token against 3.40 s for LFM2 700M. Most of the speed comes from letting some layers skip attention, so a token passes through those blocks without being compared to every earlier token. The 350M model keeps full attention in 7 of its 12 layers and the 650M model in 8 of 13, while the 1.4B model attends in all 16. Meta reports that skipping consistently beat sliding window attention, which instead limits each token to a fixed span of recent ones, and that more than three skipped layers in a row costs accuracy at the same latency, dropping TriviaQA exact match from 33.2% to 8.8%. Meta built the three models by pruning a shallow 1.8B variant of MobileLLM-Pro and inheriting its weights, which the paper says took about 35% of the tokens that pretraining MobileLLM-Pro required. Across nine reasoning and knowledge tasks Meta reports averages of 45.46, 48.57 and 55.06 for the three sizes, against 44.92, 48.52 and 50.48 for LFM2 at 350M, 700M and 1.2B, 51.79 for Gemma 3 1B and 49.01 for Llama 3.2 1B. After instruction tuning, the 1.4B model scores 47.89 on MMLU and 46.34 on HumanEval, against 49.30 and 37.80 for Llama 3.2 1B. The paper announces no release of the weights. Chart: Meta. Bubble area shows parameter count, latency measured on a Samsung Galaxy S25. Source: https://arxiv.org/abs/2603.15954 Read the article: https://llmobile.kavents.com/ticker/mobilellm-flash/MetaResearchBenchmarksAndroidPyTorchhttps://llmobile.kavents.com/ticker/mobilellm-pro/https://llmobile.kavents.com/ticker/mobilellm-pro/Mon, 10 Nov 2025 06:28:00 +0100Meta Reality Labs published the MobileLLM-Pro technical report on November 10, 2025, describing a 1.08B-parameter language model built for phones and wearables. The model handles a context window of 128,000 tokens, four times what Gemma 3 1B and Qwen 3 0.6B accept. Quantising weights and embeddings to 4 bits takes the file from 2.2 GB in bf16 down to 590 MB for the CPU build and 720 MB for the accelerator build, which cannot share embedding weights. Meta exported the model through ExecuTorch and measured it on a Samsung Galaxy S25 CPU with the XNNPACK backend and on a Samsung Galaxy S24 Hexagon Tensor Processor. On a 2,000-token prompt the paper reports 8.9 s of prefill on the CPU against 2.0 s on the Hexagon unit, and decode speeds of 33.6 tok/s and 31.6 tok/s while generating 1,000 tokens. At 8,000 tokens prefill climbs to 63.5 s on the CPU and 9.8 s on the Hexagon unit, decode falls to 19.7 and 22.8 tok/s, and the key-value cache grows from 14 MB to 40 MB. The architecture puts 30 layers of width 1280 behind 20 attention heads and 4 key-value heads, and shares one weight matrix between the input embedding and the output head, which the authors say saves 260M parameters, about a quarter of the model. Three out of every four layers use local attention, comparing each token only against the 512 most recent ones, with a global layer every fourth one and at both ends. Meta’s model card credits that interleaving with cutting the key-value cache at 8,000 tokens from 117 MB to 40 MB and with 1.8 times lower prefill latency than global attention in every layer. Meta trained the model by distilling the logits of Llama 4 Scout under a KL divergence loss across three pretraining phases on roughly 1.64T tokens of open data, the second of them a step the authors call implicit positional distillation, which stretches the context window to 128k by having the teacher pass on the ability rather than feeding the student long documents. Meta puts the base model 5.7 and 7.9 points above Gemma 3 1B and Llama 3.2 1B on average across the benchmarks its model card lists, and the paper’s table has it at 76.24 on BoolQ against 63.20 and 62.51, at 52.62 on ARC-Challenge against 38.40 and 38.28, and at 100% needle-in-a-haystack retrieval against 96.80 for Llama 3.2 1B. The instruction-tuned version scores 59.8 on HumanEval against 41.5 and 37.8, 44.8 on MMLU against Gemma’s 29.9 and Llama’s 49.3, and 62.0 on IFEval against Gemma’s 80.2. In Meta’s own human evaluation over 100 prompts per task, raters preferred Gemma 3 1B for summarisation 51 to 47 and for rewriting 49 to 45, while MobileLLM-Pro led on recall 47 to 30. Meta released four checkpoints on Hugging Face, the base model, the instruction-tuned one, and two quantisation-ready base checkpoints produced by quantisation-aware training with self-distillation from the full-precision model. One uses 4-bit group-wise weights at group size 32 with 8-bit dynamic activations and an 8-bit key-value cache for CPU backends, the other 4-bit channel-wise weights for the Apple Neural Engine and Qualcomm’s Hexagon processor, which handle group-wise quantisation poorly or not at all. Needle-in-a-haystack retrieval holds at 96.44% and 98.67% against 100% at full precision. The weights sit behind a request form and carry the FAIR Noncommercial Research License. Source: https://arxiv.org/abs/2511.06719 Read the article: https://llmobile.kavents.com/ticker/mobilellm-pro/MetaResearchDistillationQuantisationAndroidhttps://llmobile.kavents.com/ticker/executorch-1-0/https://llmobile.kavents.com/ticker/executorch-1-0/Wed, 22 Oct 2025 18:00:00 +0200The PyTorch team released ExecuTorch 1.0 on October 22, 2025, the generally available version of its runtime for PyTorch models on phones, laptops and embedded devices. It takes a model straight from PyTorch without converting it to ONNX or TFLite and without rewriting it in another language, which the team says removes a common source of numerical mismatches and lost debug information. The beta arrived in October 2024. Diagram: PyTorch documentation. Five backends moved from beta to production status. XNNPACK with Arm Kleidi covers CPU, Apple Core ML covers Apple silicon, the Qualcomm AI Engine delegate covers the Hexagon NPU, and Arm Ethos-U and Vulkan cover NPU and GPU. New backends arrived for Arm VGF, the NXP eIQ Neutron NPU, Samsung Exynos NPU and GPU, and Intel OpenVINO. Meta says features built on ExecuTorch already run on Instagram, WhatsApp, Messenger and Facebook. Anuj Kumar, Director at Reality Labs, describes what it drives in Meta’s wearables, including the Ray-Ban Display glasses with EMG band. Advanced AI features like speech recognition, motion sensing, and computer vision run on these devices by leveraging cutting-edge hardware. Lysandre Debut, Chief Open-Source Officer at Hugging Face, says more than 80 percent of the most downloaded edge-friendly language models on Hugging Face run on ExecuTorch out of the box. The team validated coverage across object detection, depth, OCR, speech recognition and segmentation as well as text and multimodal models such as Voxtral for audio and Gemma 3 for image and text input. The main branch also allows embedding ExecuTorch into native C++ desktop applications. Source: https://pytorch.org/blog/introducing-executorch-1-0/ Read the article: https://llmobile.kavents.com/ticker/executorch-1-0/MetaPyTorchDeveloper toolsNPUOpen sourcehttps://llmobile.kavents.com/ticker/mobilellm-r1/https://llmobile.kavents.com/ticker/mobilellm-r1/Fri, 12 Sep 2025 12:00:00 +0200Meta released MobileLLM-R1 on September 12, 2025, a family of reasoning models at 140M, 360M and 950M parameters that write out a chain of intermediate steps before giving an answer instead of replying directly. Meta reports the 950M model scoring 74.0 on the MATH500 maths benchmark and 19.9 on LiveCodeBench v6, against 73.0 and 14.9 for Qwen3-0.6B. The technical report followed on September 29, 2025. The three models saw 4.2T training tokens, which Meta puts at 11.7% of the 36T tokens behind Qwen3-0.6B. Pre-training ran in two phases of 2T tokens each, resampled from a curated pool of roughly 2T tokens of open datasets, followed by two mid-training phases of 100B tokens that distil from Llama-3.1-8B-Instruct as a teacher, then supervised fine-tuning on reasoning traces. Meta uses the result to argue that reasoning does not require the corpora above 10T tokens that recent models have been trained on. For the post-trained models Meta’s table puts MobileLLM-R1-950M at 74.0 on MATH500, 67.5 on GSM8K, 15.5 on AIME 2024 and 19.9 on LiveCodeBench v6, where Qwen3-0.6B reaches 73.0, 79.2, 11.3 and 14.9 and DeepSeek-R1-Distill-Qwen-1.5B reaches 83.2, 77.3, 29.1 and 19.9. Among the models whose weights, data and recipe are all public, the same AIME figure compares with 0.6 for OLMo-2 1.48B and 0.3 for SmolLM2-1.7B. On code the 950M base model scores 46.3 on HumanEval against 30.5 for Qwen3-0.6B-Base, and the 140M base model reaches 15.9 where SmolLM2-135M scores 0.0. Meta profiled the models through ExecuTorch on a Samsung Galaxy S22 with 8 GB of RAM, quantised to 4-bit weights with 8-bit dynamic activations. The 140M model generated 129.67 tok/s at a 1k context and 79.71 tok/s at 32k, the 950M model 31.05 tok/s at 1k before running out of memory at 16k, and Llama-3.2-1B 28.71 tok/s at 1k with the same memory failure. The architecture follows Meta’s earlier MobileLLM work and Llama 3.2, sharing one weight matrix between the input embedding and the output layer, and the post-trained models take a 32k context. Meta released the weights on Hugging Face together with the training code, the data sources and the data mixing ratios on GitHub, under the FAIR Noncommercial Research License, which covers research, education and analysis but not commercial use. Access to the model repositories is gated behind a form. Update, November 24, 2025. Meta released MobileLLM-R1.5 at the same three sizes, fine-tuned from the R1 models with one further round of on-policy distillation, where the student answers first and the teacher, here Llama-3.1-Nemotron-Nano-4B-v1.1, grades its own words back to it token by token. Meta reports the 950M version at 86.6 on MATH500, 82.6 on GSM8K, 39.9 on AIME 2024 and 29.1 on LiveCodeBench v6, up from 74.0, 67.5, 15.5 and 19.9 for MobileLLM-R1-950M, and says it beats DeepSeek-R1-Distill-Qwen-1.5B on every maths and coding benchmark it ran. At 360M Meta puts MATH500 at 63.4 against 28.4 and GSM8K at 52.8 against 24.5. Source: https://arxiv.org/abs/2509.24945 Read the article: https://llmobile.kavents.com/ticker/mobilellm-r1/MetaResearchBenchmarksDistillationAndroidhttps://llmobile.kavents.com/ticker/pareto-q/https://llmobile.kavents.com/ticker/pareto-q/Tue, 04 Feb 2025 19:59:00 +0100Meta published ParetoQ on February 4, 2025, a training framework that puts 1-bit, 1.58-bit, 2-bit, 3-bit and 4-bit quantisation through the same recipe so the bit widths can be compared against each other rather than against methods built for one width. The result the authors draw from it is that 1.58-bit, 2-bit and 3-bit land close together and generally beat 4-bit on accuracy against model size, while 1-bit drops away. On LLaMA-3 8B the paper’s table gives an average of 71.2 across five commonsense tasks at 2 bits and 69.0 at 1.58 bits against 74.6 for the full-precision model, then 64.1 at 1 bit. That ranking only holds when size is counted in bytes rather than parameters, which is how a phone’s memory budget works. The authors measure each model by its effective quantised size, every weight counted at its own bit width plus the embeddings at theirs, and compare across that axis. Their example is a 2-bit MobileLLM-1B scoring 61.0 on an eight-task average against 59.2 for a 4-bit MobileLLM-600M, 1.8 points higher at a smaller effective size, and the paper reports the same pattern across LLaMA models up to 8B. The paper also describes where the training behaviour changes. At 3 bits and above the fine-tuned weights stay within 10 to 20 percent of the full-precision weights they started from and recover most of the accuracy after about 10B tokens, while at 2 bits and below they move by roughly 40 percent and need around 30B tokens, which the authors read as the network rebuilding its representations instead of nudging them. On that basis they argue against training ternary models from scratch, the route taken by BitNet b1.58 and Spectra, and put their ternary 600M model at 58.7 on a six-task average, against 57.5 for the ternary 3B model of the 1-bit Era and 58.2 for Spectra’s ternary 3.9B, both computed from the same table. For speed the authors wrote a 2-bit CPU kernel and timed five MobileLLM models on an Apple M1 MacBook Pro with 32 GB of memory using 6 threads, 5 prompt tokens and 122 generated tokens, and report the 2-bit curve sitting above the 4-bit one on accuracy against decoding speed without naming a factor. The one speedup they put a number on is on an H100 NVL GPU, where their 2-bit kernel built on CUTLASS runs 4.14 times faster than FP16 and 1.24 times faster than the 4-bit Machete kernel from vLLM at a 16384 by 16384 weight shape. At 1B, 3B and 8B the authors say the in-kernel conversion overhead cancels that gain and 4-bit gives the better speed-accuracy trade-off. Hardware is also why the paper favours 2-bit over ternary despite the two sitting together on its accuracy-size front. Storing three values needs either sparsity above 90 percent to pay for the indexing or packing that complicates the matrix multiply, and the authors note that some implementations store ternary weights as 2-bit signed integers anyway, which gives up the storage saving. They write that broad adoption will need “community-wide efforts, such as INT2 support in NVIDIA tensor cores”, and the paper itself names no weight release. Update, May 30, 2025. Meta published the ParetoQ checkpoints on Hugging Face, five MobileLLM sizes from 125M to 1.5B, each as a BF16 model plus 1-bit, 1.58-bit, 2-bit, 3-bit and 4-bit versions, 30 repositories in all. They sit behind a gated form under Meta’s FAIR Noncommercial Research License, which permits research use only. The training code had gone up on GitHub on March 3, 2025. Source: https://arxiv.org/abs/2502.02631 Read the article: https://llmobile.kavents.com/ticker/pareto-q/MetaResearchQuantisationMemoryBenchmarkshttps://llmobile.kavents.com/ticker/llama-stack/https://llmobile.kavents.com/ticker/llama-stack/Wed, 25 Sep 2024 19:30:00 +0200Meta announced Llama Stack at Connect on September 25, 2024, a set of REST APIs plus client SDKs for Python, Node, Swift and Kotlin that cover the same Llama features wherever the model runs. The repository defines APIs for Inference, Safety, Memory, Agentic System, Evaluation, Post Training, Synthetic Data Generation and Reward Scoring, and lists PyTorch ExecuTorch as the on-device provider, covering inference and agents on iOS. The models announced at the same event are covered separately. Llama Stack separates providers from distributions. A provider is the implementation behind an API, which can be local code or a pointer to a remote service, and a distribution assembles providers behind one endpoint. Meta’s README states that providers can be mixed, that a hobbyist can serve a small model locally and use a cloud provider for a large one, and that the higher level APIs the app works with do not change, including across the server and mobile-device boundary. Diagram: Meta. From the Llama Stack repository documentation. On iOS the local provider is a Swift class called LocalInference, which the documentation describes as powered by ExecuTorch. It implements the same Inference interface the Swift client uses for remote servers, so an app that already streams chat completions or agent turns keeps its calling code and swaps the service it was given. Developers export the model to an ExecuTorch .pte file, bundle it with tokenizer.model in the app, and link the XNNPACK, Core ML and MPS backends plus the quantized and optimised kernels, with Meta’s own example loading a SpinQuant build of Llama 3.2 1B. The Swift package is added through Xcode’s package manager and generates its types from the Llama Stack OpenAPI spec, and Meta listed its first release on October 10, 2024. The same iOS documentation stated that on-device inference for Android was coming soon, and pointed developers to ExecuTorch’s own Android demo app in the meantime. Llama Stack, the Swift client and the Kotlin client are published under the MIT license. Update, December 6, 2024. Meta released version 0.0.54 of the Kotlin client, which adds on-device inference on Android. The README names ExecuTorch as the local inference distributor, has developers run a script that fetches a prebuilt executorch.aar with the XNNPACK delegate into the app’s libs directory, and shows the only difference between the two modes as the client builder, LlamaStackClientLocalClient with a model and tokenizer path against LlamaStackClientOkHttpClient with a base URL. The chatCompletion call that follows is identical. Source: https://github.com/meta-llama/llama-stack Read the article: https://llmobile.kavents.com/ticker/llama-stack/MetaLlamaiOSAndroidDeveloper toolshttps://llmobile.kavents.com/ticker/llama-3-2-1b-3b/https://llmobile.kavents.com/ticker/llama-3-2-1b-3b/Wed, 25 Sep 2024 19:00:00 +0200Meta released Llama 3.2 on September 25, 2024, including text-only 1B and 3B models built for phones and edge hardware. Both carry a 128K token context window and are aimed at summarisation, instruction following and rewriting that run locally, with the data staying on the device. Meta built them by structured pruning from Llama 3.1 8B, then recovered quality through knowledge distillation using logits from the 8B and 70B models during pretraining. The company reports the 3B model ahead of Gemma 2 2.6B and Phi 3.5-mini on instruction following, summarisation, prompt rewriting and tool use, and puts the 1B model on a par with Gemma. In Meta’s own table the 3B model scores 77.4 on IFEval against 61.9 for Gemma 2 2B and 59.2 for Phi-3.5-mini, and 67.0 on BFCL V2 for tool use against 27.4 and 58.4. Table: Meta. The company measured the Gemma and Phi results itself. The models shipped with day-one support for Qualcomm and MediaTek silicon and run on Arm, which Meta says covers 99 percent of mobile devices. Weights are on llama.com and Hugging Face, with deployment paths through PyTorch ExecuTorch for devices and Ollama for single-node setups, and the company lists more than 25 partner platforms at launch. Source: https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ Read the article: https://llmobile.kavents.com/ticker/llama-3-2-1b-3b/MetaLlamaOpen weightsQualcommMediaTekhttps://llmobile.kavents.com/ticker/mobile-llms/https://llmobile.kavents.com/ticker/mobile-llms/Thu, 22 Feb 2024 19:58:00 +0100Meta Reality Labs researchers published MobileLLM on February 22, 2024, a family of language models built to run on a phone rather than in a data centre. The paper argues that below a billion parameters the shape of the network matters more than the parameter count or the amount of training data. Its 125M and 350M models beat the previous best models at those sizes by 2.7 and 4.3 points on eight zero-shot common sense reasoning tasks. The central finding is that depth beats width. The authors trained 19 models of roughly equal size but different proportions, 9 around 125M parameters and 10 around 350M, and report that 30-layer and 42-layer designs scored higher than the 12-layer shape used by OPT, GPT-Neo and Pythia at the same size. Sharing one weight matrix between the input embedding and the output layer removes 16M parameters, about 11.8% of a 125M model, for a 0.2 point accuracy drop the authors then recover by spending the freed budget on two more layers. Grouped query attention, where several query heads read one shared set of key and value heads, cuts 16 key-value heads to 4 at close to the same accuracy. The paper sets out what a phone can actually spare. The authors put DRAM on current handsets at 6 to 12 GB, shared with the operating system and every other app, and argue that an app should not claim more than 10% of it. For energy they use a rule of thumb of 0.1 J per token per billion parameters, which puts a 7B model at 0.7 J/token, drains 0.2% of an iPhone battery every 64 tokens and allows under 2 hours of conversation at 10 tok/s. A 350M model with 8-bit weights costs 0.035 J/token by the same arithmetic, which the authors say covers a full day of use. Meta also profiled the models on hardware rather than estimating. Running the 125M model in FP16 through ExecuTorch on an iPhone 13 with iOS 17.2.1 and the Metal Performance Shaders backend, the authors measured 39.2 ms to load, 1361.7 ms to initialise and 15.6 ms to execute, the last averaged over 50 runs. MobileLLM-LS, the layer-sharing variant, computes each transformer block twice in a row so that 30 blocks of weights behave like 60 layers at the same file size, a pattern the team picked over the alternatives because the repeated weights stay in the roughly 20 MB of on-chip cache instead of being fetched from DRAM again. That costs 2.2% more loading and initialisation time and 2.6% more execution time, against 143% and 86% for a model that really has 60 layers, and adds 0.7 and 0.8 points of accuracy at 125M and 350M. After fine-tuning, MobileLLM-350M scores 3.28 on the MT-Bench chat benchmark against 1.37 for OPT-350M and 2.24 for the 1.3B OPT, and the layer-shared 350M wins 48.2% of AlpacaEval comparisons against text-davinci-001, a model that wins 50% against itself. For API calling the team built a synthetic set of 5,000 training and 2,500 test conversations that turn requests such as setting an alarm into a function call, and MobileLLM-350M scored 65.3 on intent exact match against 62.8 for Llama 2 7B, while trailing the larger model on the structure of the call and on the wording of the reply. Quantising weights and activations to 8 bits costs under 0.5 points. The work was published at ICML 2024 and the code is in Meta’s repository. Source: https://arxiv.org/abs/2402.14905 Read the article: https://llmobile.kavents.com/ticker/mobile-llms/MetaResearchMemoryiPhoneBenchmarks