MobileVLM V2 runs 1.7B, 3B and 7B vision models on 144 image tokens
Researchers at Meituan, Zhejiang University and Dalian University of Technology published MobileVLM V2 on February 6, 2024, a family of vision language models at 1.7B, 3B and 7B parameters, each pairing a CLIP ViT-L/14 encoder that reads images at 336 by 336 pixels with a MobileLLaMA or Vicuna language model. Averaged over six benchmarks covering visual question answering, science questions, reading text in images, object hallucination and two multimodal test suites, the authors score the 1.7B model at 64.2 against 62.7 for LLaVA-1.5 3.3B, and the 3B model at 68.1 against 66.7 for MoE-LLaVA-2.7B×4, a mixture of experts where only part of the model is active per token. Their 7B model reaches 72.1, ahead of ShareGPT4V 7B at 70.8 and LLaVA-1.5 7B at 68.8 on the same average.
The authors measured generation over 256 output tokens on an NVIDIA AGX Jetson Orin developer board, not on a phone, with the language model quantised to 4-bit and run through llama.cpp. They report 51.63 tok/s for the 1.7B model, 30.80 tok/s for the 3B and 15.49 tok/s for the 7B, against 43.39, 20.45 and 12.96 tok/s for the three LLaVA-1.5 sizes in the same table. The authors state they wrote their own CUDA implementation of the new projector for that board.
Most of the speed comes from how much of the image reaches the language model. The encoder turns a 336 by 336 picture into 576 patch tokens, and the projector between the two models, which the authors call LDPv2, first runs two 1 by 1 convolutions to match the language model’s width. It then averages every 2 by 2 block of neighbouring patches into a single token, leaving 144, and finally adds a small depthwise convolution back onto the result so each surviving token keeps a sense of where it sat in the picture. That positional part holds 0.02M parameters against 12.64M for the equivalent block in the first MobileVLM, and the whole projector 6.32M against 18.94M, while the six-benchmark average in the authors’ ablation rises from 62.8 to 64.2.
Dropping tokens costs accuracy on fine detail. The authors removed the pooling step from the 7B model, which leaves all 576 tokens and puts it at the same latency as ShareGPT4V, and the six-benchmark average rises from 72.1 to 73.5, with most of the gain on TextVQA, a reading task full of small objects. vivo AI Lab took the same 2 by 2 merge further nine months later in BlueLM-V-3B, cutting 729 tokens per image tile to 196 and measuring the result on a phone NPU.
The rest of the gain over the first MobileVLM comes from data and from what stays trainable. The authors pretrain on 1.2 million captioned images from ShareGPT4V, then run a second stage over 2.4 million samples drawn from Visual Dialog, TextVQA, COCO Caption, SBU and other academic sets, 3.6 million in total, and they train the projector together with the language model in both stages while the vision encoder stays frozen, where LLaVA-1.5 and the first MobileVLM keep the language model fixed during pretraining. Both stages together took 8 NVIDIA A100 GPUs for about 14 hours. Meituan put weights for all three sizes on Hugging Face the same day, and the code is published under the Apache 2.0 license.