<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Nvidia · LLMobile.news</title><link>https://llmobile.kavents.com/tags/nvidia/</link><description>A concise news ticker covering AI on mobile devices, local models, apps and hardware.</description><language>en-GB</language><atom:link href="https://llmobile.kavents.com/tags/nvidia/index.xml" rel="self" type="application/rss+xml"/><item><title>Nemotron-Flash-1B decodes 1.9 times faster than Qwen3-0.6B on an H100</title><link>https://llmobile.kavents.com/ticker/nemotron-flash/</link><guid isPermaLink="true">https://llmobile.kavents.com/ticker/nemotron-flash/</guid><pubDate>Mon, 24 Nov 2025 09:46:00 +0100</pubDate><description>Nvidia published Nemotron-Flash on November 24, 2025, a family of small language models built around measured decoding latency instead of parameter count. Nvidia reports Nemotron-Flash-1B decoding 8k tokens in 14.45 seconds on a single Nvidia H100 at batch size 1, against 27.55 seconds for Qwen3-0.6B, while averaging 49.63 across 16 benchmarks to Qwen3-0.6B&amp;amp;rsquo;s 44.11. The paper was accepted at NeurIPS 2025.
The models mix three kinds of layer rather than using attention throughout. Nvidia interleaves DeltaNet and Mamba2 layers, operators that carry a fixed-size state instead of a cache that grows with the context, with a small number of ordinary attention layers, and an evolutionary search picked which operator goes where. Nemotron-Flash-1B holds 0.96B parameters in 12 blocks with only 2 full attention layers, and Nemotron-Flash-3B holds 2.7B parameters in 18 blocks with 3. Nvidia built both wider and shallower than other models of the same size, where Qwen3-0.6B stacks 28 layers, and writes that deep and thin models are parameter efficient but sit off the accuracy-latency frontier.
Nemotron-Flash-3B averages 60.98 across the same 16 tasks, which cover MMLU, commonsense reasoning, math, coding and recall, against 58.96 for Qwen2.5-3B and 55.47 for Qwen3-1.7B. On the same H100, with all models running through TensorRT-LLM AutoDeploy kernels and CUDA Graph, Nvidia measured 28.71 seconds for the 3B against 49.40 for Qwen2.5-3B and 36.20 for Qwen3-1.7B, so 1.7 and 1.3 times lower latency. Throughput was measured separately, with a 32k-token input at the largest batch size that still fits in memory, and Nvidia reports 2939 tok/s for the 3B against 459 and 157 for the two Qwen models, and 7289 tok/s for the 1B against 160 for Qwen3-0.6B. Every figure here is Nvidia&amp;amp;rsquo;s own, taken on data centre GPUs, and the paper gives no phone or embedded measurements, with the earlier depth and operator studies run on an Nvidia A100.
Nvidia trained both models on 4.5T tokens using 256 H100 GPUs and released three checkpoints on Hugging Face, Nemotron-Flash-1B, the 3B base model and an instruction-tuned Nemotron-Flash-3B-Instruct, which averages 52.84 across MMLU, GPQA, GSM8K and IFEval against 48.17 for Qwen2.5-1.5B. The model cards set the licence to Creative Commons Attribution-NonCommercial 4.0, so the weights are downloadable but not licensed for commercial use, and neither Apache nor an Nvidia open model licence applies. Running them needs the TensorRT-LLM AutoDeploy attention kernel alongside the Mamba2 and FlashLinearAttention implementations, since Nvidia states that vLLM and TensorRT-LLM do not support DeltaNet-style linear attention on their own.
Chart: Fu et al., Figure 1 of the paper. Nvidia&amp;amp;#39;s own measurements on an Nvidia H100.
Source: https://arxiv.org/abs/2511.18890
Read the article: https://llmobile.kavents.com/ticker/nemotron-flash/</description><category>Nvidia</category><category>Research</category><category>Benchmarks</category><category>Open weights</category></item><item><title>Intelligence per watt puts local model coverage at 88.7% of real queries</title><link>https://llmobile.kavents.com/ticker/intelligence-per-watt/</link><guid isPermaLink="true">https://llmobile.kavents.com/ticker/intelligence-per-watt/</guid><pubDate>Tue, 11 Nov 2025 07:33:00 +0100</pubDate><description>Researchers at Stanford University and Together AI profiled more than 20 local models on 1M real queries and report that at least one model with 20B or fewer active parameters answered 88.7% of them correctly. The best single model in their set, GPT-OSS-120B, reached 71.4% on its own, averaged across the four query collections. Their 2025 efficiency figure comes from that model running on an Apple M4 Max.
The paper proposes intelligence per watt, mean task accuracy divided by the mean power the accelerator draws while answering, so a pairing that gets more answers right on the same power budget scores higher. The best local pairing each year moved from 7.92×10⁻⁴ accuracy per watt for Mixtral 8x7B on an NVIDIA Quadro RTX 6000 in 2023, to 1.80×10⁻³ for Llama 3.1 8B on an RTX 6000 Ada in 2024, to 4.18×10⁻³ for GPT-OSS-120B on an Apple M4 Max in 2025, which the authors put at a 5.3 times gain over two years. Query coverage for that single best pairing rose from 23.2% to 71.3% across the same three years. A companion metric divides accuracy by the energy a query consumes instead, so slow generation counts against a device.
Diagram: Saad-Falcon et al., Figure 1 of the paper, licensed CC BY 4.0. The hardware panel separates the accelerators the authors treat as local from the cloud parts. The 1M queries come from four collections. The team took 500K prompts from WildChat, a month of real ChatGPT traffic, 500K from NaturalReasoning, 12K from MMLU Pro and 26.5K from SuperGPQA, all single-turn. WildChat has no ground truth, so GPT-4o judged whether a local model&amp;amp;rsquo;s answer matched one written by Qwen3-235B, while the other three collections supplied reference answers. Every run used a batch size of 1, one query at a time.
The hardware the authors call local is desktop and workstation class rather than mobile. Their 2025 numbers ran on an Apple Mac Studio with an M4 Max, 128 GB of unified memory and 480 W of continuous power, and the 2023 and 2024 points used NVIDIA workstation cards. No phone, tablet or wearable was profiled. Running the same Qwen3 models, the M4 Max reached 1.40 times lower accuracy per watt than an NVIDIA B200 and 1.6 to 2.3 times lower accuracy per joule, a gap the authors read as headroom for local accelerator design.
For the savings case the team simulated 80.2M queries over 24 hours split between four small models on M4 Max machines and Qwen3-235B in the cloud. Perfect routing to the smallest capable model cut energy by 80.4% against sending everything to the frontier model, and a router that picks correctly 80% of the time cut it by 64.3%, with the perfect case described as a theoretical maximum. The authors note that coverage falls in technical domains such as architecture and engineering, that GPT-OSS-120B answered 88.9% of WildChat queries against 64.9% of NaturalReasoning ones, and that they release their profiling harness so the measurements can be repeated on new hardware.
Source: https://arxiv.org/abs/2511.07885
Read the article: https://llmobile.kavents.com/ticker/intelligence-per-watt/</description><category>Research</category><category>Benchmarks</category><category>Apple Silicon</category><category>Nvidia</category><category>Qwen</category></item><item><title>D2MoE picks a bit-width per token, 1.39 times the throughput at up to 53 percent less memory</title><link>https://llmobile.kavents.com/ticker/d2moe/</link><guid isPermaLink="true">https://llmobile.kavents.com/ticker/d2moe/</guid><pubDate>Thu, 17 Apr 2025 07:37:35 +0200</pubDate><description>Researchers at the Hong Kong University of Science and Technology and Shenzhen University published D2MoE on April 17, 2025, a serving system for mixture-of-experts models on edge hardware, accepted at ACM MobiCom 2025. On a laptop with a 6 GB NVIDIA RTX 3060 and on a Jetson AGX Orin with 64 GB, the authors measure Mixtral 8x7B at 1.14 to 1.39 times the throughput of EdgeMoE and LLaMA-MoE-3.5B at 1.06 to 1.16 times, the second one while using 33 to 53 percent less memory. Against MoQE-DynaIO, which quantises every expert to one fixed bit-width and loads them on demand, they report 1.42 to 3.37 times the throughput.
A mixture of experts replaces the feedforward block in each layer with a set of small networks, and a router sends every token to only a few of them, so the work per token stays small while every expert still has to be held in memory or fetched from somewhere. The paper puts Mixtral 8x7B at over 90 GB in float16, with experts accounting for 89.9 percent of the parameters, against the 6 GB the RTX 3060 laptop offers. One answer is to keep the whole expert set small enough to stay resident, the way Ai2 sizes OLMoE and Meta AI sizes MobileMoE. D2MoE takes the other one, quantising experts hard and loading them from lower tiers on demand, which is also how Edge0 fits a large model into a small budget.
That trade turns a memory problem into a transfer problem. On the RTX 3060 the authors time a single LLaMA-MoE expert at 3.1 ms of computation against roughly 20 ms of data transfer, so the GPU sits idle waiting for weights, and holding several quantised copies of each expert to choose from would give the memory back. D2MoE therefore routes each token twice, once to an expert and once to a bit-width for that expert, and nests the quantised weights so that a wider one contains the narrower ones, which the authors name after matryoshka dolls. They note that 4-bit LLaMA-MoE experts quantised with llama.cpp take 3.81 GB, while storing separate 2-bit, 3-bit and 4-bit copies takes 9.62 GB.
A scheduler then orders expert loads hottest first so that transfers overlap computation inside a set memory budget, and the paper puts the added cost of the second router at 0.28 percent of computation, 0.53 percent of memory and 1.67 percent of latency for LLaMA-MoE-3.5B. On WikiText2 the configuration meant to match 8-bit experts scores 4.09 perplexity on Mixtral 8x7B, against 4.04 for the baseline that holds all 8-bit experts in memory and 4.38 for EdgeMoE. Setting a model up is offline work, at about 2 hours of router fine-tuning plus 10 minutes of quantisation for LLaMA-MoE-3.5B and over 4 hours plus 20 minutes for Mixtral 8x7B. The prototype is roughly 2,500 lines of Python and CUDA on top of PyTorch, written for NVIDIA Ampere and Ada Lovelace, and the paper names no code release and reports no measurements on phones.
Chart: Wang et al. The authors&amp;amp;#39; own measurements on the RTX 3060 laptop, where RN is the number of concurrent requests. Each baseline appears as a single point at the budget it needs, while D2MoE is plotted across budgets.
Source: https://arxiv.org/abs/2504.15299
Read the article: https://llmobile.kavents.com/ticker/d2moe/</description><category>Mixture of experts</category><category>Memory</category><category>Quantisation</category><category>Nvidia</category><category>Research</category></item><item><title>Raspberry Pi 5 beats human reading speed only up to Phi 3.5 mini</title><link>https://llmobile.kavents.com/ticker/on-device-inference-tradeoffs/</link><guid isPermaLink="true">https://llmobile.kavents.com/ticker/on-device-inference-tradeoffs/</guid><pubDate>Wed, 12 Mar 2025 08:01:00 +0100</pubDate><description>Maximilian Abstreiter, Sasu Tarkoma and Roberto Morabito of the University of Helsinki and EURECOM measured 11 language models from 0.5B to 9.2B parameters on a Raspberry Pi 5 and an NVIDIA Jetson Orin Nano Developer Kit, both with 8 GB of RAM, running every model through llama.cpp at 4-bit. On the Pi, the authors report that only models up to Llama 3.2 3B, or up to Phi 3.5 mini in the faster of the two 4-bit formats they tested, generated text above 5.3 tokens per second, the rate they derive from average English reading speed. The Orin&amp;amp;rsquo;s GPU stayed above that rate for all 11 models, its CPU cores up to InternLM 2.5 7B in that same format.
Generation speed drops off steeply with model size in the paper&amp;amp;rsquo;s charts. Qwen 2 0.5B runs at roughly 46 tokens per second on the Orin&amp;amp;rsquo;s CPU cores and roughly 33 on the Pi, while Phi 3.5 mini at 3.8B parameters sits at about 5 on the Pi, about 10 on the Orin CPU and about 15 on the Orin GPU. The authors also contradict Meta&amp;amp;rsquo;s claim that Llama 3.2 is optimised for Arm processors, since the advantage they saw for Llama 3.2 1B showed up in GPU inference and not on either CPU.
Memory is where the measurements turn hardest. Peak usage grows close to linearly with model size, reaching about 7.9 GB for Gemma 2 9B on the Pi, and it flattens out on the Orin above Phi 3.5 mini because under 7 GB of the nominal 8 GB is usable once the operating system and background processes take their more than 1 GB. Raising the context window to 1028 tokens pushed the largest model into constant page faults on the CPU and made the Orin crash and restart during GPU inference. First-run load times ran from 5 seconds to over 200 seconds, and for Yi 1.5 6B and up the faster 4-bit format raised load times by 17 to 53 times, because those models get evicted from memory and re-read from the microSD card for every inference.
Scaled to the 18.84 Wh battery of a Samsung Galaxy S24 Ultra at full utilisation, the March version of the paper puts Gemma 2 9B at 193 inference runs on the Pi and 101 on the Orin GPU, against 3,928 and 4,280 runs for Qwen 2 0.5B, with no configuration lasting 7 hours. The setup was built to keep thermal throttling out of the results rather than to observe it, so the authors fitted the Pi with an add-on heatsink after it overheated in early experiments, ran the fans on both boards at full speed and left 30 seconds between runs.
Quantisation cost measurable quality. Perplexity, which tracks how uncertain a model is about its next token, rose against the 16-bit weights for every model and both 4-bit formats and rose most sharply for the smallest models, while the downstream benchmarks showed almost no accuracy drop. The paper is on arXiv and under review at an ACM journal.
Update, November 20, 2025. The authors revised the paper after validating their power measurements against external hardware, which raised the measured consumption. The battery estimate for Gemma 2 9B falls from 193 runs to 75 on the Pi and from 101 to 73 on the Orin GPU, and no configuration now reaches 3.5 hours at full utilisation. The speed, memory and quality results are unchanged.
Source: https://arxiv.org/abs/2503.09114
Read the article: https://llmobile.kavents.com/ticker/on-device-inference-tradeoffs/</description><category>Research</category><category>Benchmarks</category><category>llama.cpp</category><category>Quantisation</category><category>Nvidia</category></item><item><title>MobiLLM fine-tunes OPT-1.3B in 4.5 GB on device by moving backpropagation to a server</title><link>https://llmobile.kavents.com/ticker/mobillm-side-tuning/</link><guid isPermaLink="true">https://llmobile.kavents.com/ticker/mobillm-side-tuning/</guid><pubDate>Thu, 27 Feb 2025 08:58:02 +0100</pubDate><description>Researchers at Pengcheng Laboratory, the University of Houston, Stevens Institute of Technology, Keio University and the University of Waterloo posted MobiLLM on arXiv on February 27, 2025, a way to fine-tune a language model on a single device while a server does the training work. Their OPT-1.3B table puts the device-side memory at 4.50 GB, against 14.57 GB for LoRA, 13.37 GB for BitFit, 10.80 GB for the LST side-tuning baseline and 20.90 GB for full fine-tuning, all at a batch size of 16 and a sequence length of 256. The device is an NVIDIA Jetson Xavier NX whose GPU can use 4.6 GB of the board&amp;amp;rsquo;s 8 GB of shared RAM, so MobiLLM is the only method in the authors&amp;amp;rsquo; comparison that fits a billion-parameter fine-tune onto it. For OPT-350M the same table gives 1.64 GB against 6.70 GB for LoRA and 7.91 GB for full fine-tuning, which is 0.01 GB more than running that model for inference on the device.
Side tuning puts the two halves of training in two places. The device keeps the pre-trained backbone frozen and runs only the forward pass through it, while the server holds a small trainable side network built from adapter modules, each one a projection down to a narrow dimension, a non-linear step and a projection back up. The adapters sit parallel to the backbone rather than inside it, so gradients never travel back through the frozen model, and the authors trace the saving to the device storing neither the per-layer activations that backpropagation would need nor any optimizer state. It is a different bargain from splitting one model across several nearby devices, as Confidant does, which the authors argue needs a stable set of peers and leaves no single device holding a model it can run on its own.
What crosses the network is one-way and happens only during the forward pass. The device quantises the intermediate activations of selected backbone blocks, by default to 4 bits, and sends them to the server along with the labels for the batch and metadata such as batch indices, and nothing travels back while training runs. Raw samples stay on the device, which is the authors&amp;amp;rsquo; privacy argument, though the labels do leave it, and the paper states that the activation outputs of each block distort the input embeddings enough to make the original samples hard to infer, without reporting a reconstruction attack that tests this. Quantising those activations cuts what has to be sent per iteration for OPT-1.3B from 400 MB to 100.2 MB, at a GLUE average of 81.3 in the NF4 format against 81.5 unquantised.
The authors put the speed gain at 1.8 times to a target test accuracy on the Xavier and 2.3 times on a CPU-only Huawei MateBook laptop with a Core i5-13500H, both measured against LoRA, over a Wi-Fi 5 link to a server with an NVIDIA A100. Their link-rate table for OPT-350M is the more direct measurement, putting one iteration at a batch size of 16 at 7.48 s over 60 Mbps against 8.4 s for training on the device alone, and at 9.8 s when the uplink drops to 10 Mbps, where they say transmission becomes the bottleneck. Accuracy is the cost they name, with MobiLLM averaging 81.3 across the eight GLUE tasks on OPT-1.3B against 83.6 for full fine-tuning and 83.0 for LoRA, and 78.8 against 80.7 on OPT-350M.
Fine-tuning this way still needs a reachable server, and the authors point at 5G and 6G base stations and home Wi-Fi hubs as where one would sit rather than testing either. They name two next steps, quantising the frozen backbone weights themselves, for which a side experiment puts device-side memory at 2.751 GB for OPT-1.3B at INT4 precision against a 2 percent accuracy drop and at 1.176 GB for OPT-350M, and filtering which activations get sent at all to cut the upload further. The paper is posted under the arXiv non-exclusive distribution license rather than a Creative Commons one, and it names no code release.
Source: https://arxiv.org/abs/2502.20421
Read the article: https://llmobile.kavents.com/ticker/mobillm-side-tuning/</description><category>Research</category><category>Memory</category><category>Quantisation</category><category>Nvidia</category></item><item><title>Apple team finds H100 last on tokens per dollar for models up to 2B</title><link>https://llmobile.kavents.com/ticker/small-model-training-bottlenecks/</link><guid isPermaLink="true">https://llmobile.kavents.com/ticker/small-model-training-bottlenecks/</guid><pubDate>Fri, 25 Oct 2024 12:30:00 +0200</pubDate><description>Seven Apple researchers measured what it costs to train language models up to 2B parameters on rented Nvidia GPUs and found the most expensive card was the worst buy. The H100-80GB returned the fewest tokens per dollar of the three cards tested at every one of their four model sizes, 100M, 500M, 1B and 2B, while costing 2.53 times as much per hour as the A100-40GB.
The price scale comes from normalising published hourly rates at Lambda Labs and Google Cloud, with the A100-40GB set at 1.00 and the A100-80GB landing at 1.46. The same comparison shows cost efficiency falling as GPUs are added, with a single card the best or near-best point at all four model sizes. For the 1B model, tokens per dollar drop from about 17,000 on one GPU to about 10,300 on 64.
FlashAttention mattered more at small scale than at large. Reading the authors&amp;amp;rsquo; first figure, the 100M model gained roughly three times the tokens per dollar over vanilla attention, while at 2B the gain was under two times. The authors put this down to the cost of attention being quadratic in context length, so it dominates once the hidden dimension shrinks and the small models become bound by data movement between CPU and GPU and between GPUs. Vanilla attention also ran out of memory at a global batch size of 1024 on the 1B and 2B models, where FlashAttention did not.
Two habits carried over from large-model training did not pay off. Plain Distributed Data Parallel, which keeps a full copy of the model on every GPU, beat both sharded FSDP variants for the smaller models because it moves less data, and FSDP only pulled ahead at 2B, where it kept training at a global batch size that made DDP run out of memory. Filling the cards was not cost-optimal either, since the batch size figure peaks around a global batch size of 128 for the 100M model, 32 for 500M, 16 for 1B and 8 for 2B, in every case well short of what the memory allowed.
These are training runs on datacentre GPUs, not anything running on a phone, and the authors frame the work as guidance for labs with small budgets rather than for deployment. They name their own limits, running each setup at least three times for only 10 training steps, reporting normalised price ratios rather than absolute dollars, using no gradient accumulation, and leaving GPU utilisation and memory bandwidth measurements to future work. They also assume that settings such as the learning rate can be retuned to converge at whichever hardware configuration turns out cheapest. The 8-page paper is free to read on arXiv under the site&amp;amp;rsquo;s non-exclusive distribution licence.
Source: https://arxiv.org/abs/2410.19456
Read the article: https://llmobile.kavents.com/ticker/small-model-training-bottlenecks/</description><category>Apple</category><category>Nvidia</category><category>Research</category><category>Benchmarks</category><category>PyTorch</category></item><item><title>MeRino designs sub-100M language models that run 4.9 times faster than OPT-350M</title><link>https://llmobile.kavents.com/ticker/merino-iot-language-models/</link><guid isPermaLink="true">https://llmobile.kavents.com/ticker/merino-iot-language-models/</guid><pubDate>Wed, 28 Feb 2024 04:20:00 +0100</pubDate><description>Researchers at the University of Central Florida and the University of Technology Sydney published MeRino on February 28, 2024, a method that draws up a transformer architecture to fit a given device&amp;amp;rsquo;s limits on parameters, arithmetic and latency, instead of shrinking an existing model to fit. Their largest design, MeRino-64M, reaches the same 0.408 average accuracy across twelve zero-shot tasks as Meta&amp;amp;rsquo;s OPT-350M while carrying 64M parameters against 331M. The authors time it at 114 ms against 559 ms for a 128-token input on an NVIDIA Jetson Nano, or 4.9 times faster.
The search scores a candidate architecture without ever training it. Each candidate shape, meaning how many blocks it has and how wide and how deep each block is, gets a single number that the authors compute from the dimensions of its attention and feedforward matrices and treat as a measure of how much information the network can represent, with a second term that penalises shapes which are deep and narrow, because those train badly. An evolutionary algorithm mutates block widths and depths for 100,000 rounds, discards everything that breaks the budget, keeps the highest-scoring survivors and returns the best one. A precomputed lookup table stands in for the matrix decomposition the score would otherwise need, and the authors put its average error at 0.03 percent.
Because the score is arithmetic over an architecture&amp;amp;rsquo;s dimensions rather than a forward pass, the search runs on the target board itself. The paper puts one MeRino search at 0.05 hours and 0.75 Wh on the Jetson Nano&amp;amp;rsquo;s CPU, against 1.2 hours and 300 Wh for the TE-NAS architecture search on an NVIDIA GTX 1080Ti. Every device measurement in the paper comes from that Jetson Nano developer board, taken at batch size 1 and sequence length 128 and averaged over 16 runs, and the authors report no measurements on phones or tablets. Training the designed models is separate and conventional work, at 600,000 steps on 8 NVIDIA H100 GPUs.
The same tables show MeRino-64M behind OPT-350M on language modelling, at 22.47 against 18.51 perplexity on WikiText-2 and 27.06 against 23.08 on Penn Treebank, where lower is better. The smallest design, MeRino-52M, matches Pythia-70M at 0.383 average zero-shot accuracy and 48 ms against 95 ms. The authors state that the weights and evaluation for Meta&amp;amp;rsquo;s MobileLLM were not released when they submitted, so they do not compare against it, and they report that 8-bit weight quantisation leaves accuracy almost unchanged while 4-bit costs about 1 percent. The work was accepted at AAAI 2025 and the paper names no code release.
Source: https://arxiv.org/abs/2403.07921
Read the article: https://llmobile.kavents.com/ticker/merino-iot-language-models/</description><category>Research</category><category>Benchmarks</category><category>Nvidia</category><category>Memory</category></item><item><title>MobileVLM V2 runs 1.7B, 3B and 7B vision models on 144 image tokens</title><link>https://llmobile.kavents.com/ticker/mobilevlm-v2/</link><guid isPermaLink="true">https://llmobile.kavents.com/ticker/mobilevlm-v2/</guid><pubDate>Tue, 06 Feb 2024 08:16:36 +0100</pubDate><description>Researchers at Meituan, Zhejiang University and Dalian University of Technology published MobileVLM V2 on February 6, 2024, a family of vision language models at 1.7B, 3B and 7B parameters, each pairing a CLIP ViT-L/14 encoder that reads images at 336 by 336 pixels with a MobileLLaMA or Vicuna language model. Averaged over six benchmarks covering visual question answering, science questions, reading text in images, object hallucination and two multimodal test suites, the authors score the 1.7B model at 64.2 against 62.7 for LLaVA-1.5 3.3B, and the 3B model at 68.1 against 66.7 for MoE-LLaVA-2.7B×4, a mixture of experts where only part of the model is active per token. Their 7B model reaches 72.1, ahead of ShareGPT4V 7B at 70.8 and LLaVA-1.5 7B at 68.8 on the same average.
The authors measured generation over 256 output tokens on an NVIDIA AGX Jetson Orin developer board, not on a phone, with the language model quantised to 4-bit and run through llama.cpp. They report 51.63 tok/s for the 1.7B model, 30.80 tok/s for the 3B and 15.49 tok/s for the 7B, against 43.39, 20.45 and 12.96 tok/s for the three LLaVA-1.5 sizes in the same table. The authors state they wrote their own CUDA implementation of the new projector for that board.
Chart: Chu et al., measured on an NVIDIA Jetson Orin with llama.cpp. Most of the speed comes from how much of the image reaches the language model. The encoder turns a 336 by 336 picture into 576 patch tokens, and the projector between the two models, which the authors call LDPv2, first runs two 1 by 1 convolutions to match the language model&amp;amp;rsquo;s width. It then averages every 2 by 2 block of neighbouring patches into a single token, leaving 144, and finally adds a small depthwise convolution back onto the result so each surviving token keeps a sense of where it sat in the picture. That positional part holds 0.02M parameters against 12.64M for the equivalent block in the first MobileVLM, and the whole projector 6.32M against 18.94M, while the six-benchmark average in the authors&amp;amp;rsquo; ablation rises from 62.8 to 64.2.
Diagram: Chu et al., MobileVLM V2. Dropping tokens costs accuracy on fine detail. The authors removed the pooling step from the 7B model, which leaves all 576 tokens and puts it at the same latency as ShareGPT4V, and the six-benchmark average rises from 72.1 to 73.5, with most of the gain on TextVQA, a reading task full of small objects. vivo AI Lab took the same 2 by 2 merge further nine months later in BlueLM-V-3B, cutting 729 tokens per image tile to 196 and measuring the result on a phone NPU.
The rest of the gain over the first MobileVLM comes from data and from what stays trainable. The authors pretrain on 1.2 million captioned images from ShareGPT4V, then run a second stage over 2.4 million samples drawn from Visual Dialog, TextVQA, COCO Caption, SBU and other academic sets, 3.6 million in total, and they train the projector together with the language model in both stages while the vision encoder stays frozen, where LLaVA-1.5 and the first MobileVLM keep the language model fixed during pretraining. Both stages together took 8 NVIDIA A100 GPUs for about 14 hours. Meituan put weights for all three sizes on Hugging Face the same day, and the code is published under the Apache 2.0 license.
Source: https://arxiv.org/abs/2402.03766
Read the article: https://llmobile.kavents.com/ticker/mobilevlm-v2/</description><category>Research</category><category>Benchmarks</category><category>llama.cpp</category><category>Open weights</category><category>Nvidia</category></item></channel></rss>