MobiLLM fine-tunes OPT-1.3B in 4.5 GB on device by moving backpropagation to a server

Researchers at Pengcheng Laboratory, the University of Houston, Stevens Institute of Technology, Keio University and the University of Waterloo posted MobiLLM on arXiv on February 27, 2025, a way to fine-tune a language model on a single device while a server does the training work. Their OPT-1.3B table puts the device-side memory at 4.50 GB, against 14.57 GB for LoRA, 13.37 GB for BitFit, 10.80 GB for the LST side-tuning baseline and 20.90 GB for full fine-tuning, all at a batch size of 16 and a sequence length of 256. The device is an NVIDIA Jetson Xavier NX whose GPU can use 4.6 GB of the board’s 8 GB of shared RAM, so MobiLLM is the only method in the authors’ comparison that fits a billion-parameter fine-tune onto it. For OPT-350M the same table gives 1.64 GB against 6.70 GB for LoRA and 7.91 GB for full fine-tuning, which is 0.01 GB more than running that model for inference on the device.

Side tuning puts the two halves of training in two places. The device keeps the pre-trained backbone frozen and runs only the forward pass through it, while the server holds a small trainable side network built from adapter modules, each one a projection down to a narrow dimension, a non-linear step and a projection back up. The adapters sit parallel to the backbone rather than inside it, so gradients never travel back through the frozen model, and the authors trace the saving to the device storing neither the per-layer activations that backpropagation would need nor any optimizer state. It is a different bargain from splitting one model across several nearby devices, as Confidant does, which the authors argue needs a stable set of peers and leaves no single device holding a model it can run on its own.

What crosses the network is one-way and happens only during the forward pass. The device quantises the intermediate activations of selected backbone blocks, by default to 4 bits, and sends them to the server along with the labels for the batch and metadata such as batch indices, and nothing travels back while training runs. Raw samples stay on the device, which is the authors’ privacy argument, though the labels do leave it, and the paper states that the activation outputs of each block distort the input embeddings enough to make the original samples hard to infer, without reporting a reconstruction attack that tests this. Quantising those activations cuts what has to be sent per iteration for OPT-1.3B from 400 MB to 100.2 MB, at a GLUE average of 81.3 in the NF4 format against 81.5 unquantised.

The authors put the speed gain at 1.8 times to a target test accuracy on the Xavier and 2.3 times on a CPU-only Huawei MateBook laptop with a Core i5-13500H, both measured against LoRA, over a Wi-Fi 5 link to a server with an NVIDIA A100. Their link-rate table for OPT-350M is the more direct measurement, putting one iteration at a batch size of 16 at 7.48 s over 60 Mbps against 8.4 s for training on the device alone, and at 9.8 s when the uplink drops to 10 Mbps, where they say transmission becomes the bottleneck. Accuracy is the cost they name, with MobiLLM averaging 81.3 across the eight GLUE tasks on OPT-1.3B against 83.6 for full fine-tuning and 83.0 for LoRA, and 78.8 against 80.7 on OPT-350M.

Fine-tuning this way still needs a reachable server, and the authors point at 5G and 6G base stations and home Wi-Fi hubs as where one would sit rather than testing either. They name two next steps, quantising the frozen backbone weights themselves, for which a side experiment puts device-side memory at 2.751 GB for OPT-1.3B at INT4 precision against a 2 percent accuracy drop and at 1.176 GB for OPT-350M, and filtering which activations get sent at all to cut the upload further. The paper is posted under the arXiv non-exclusive distribution license rather than a Creative Commons one, and it names no code release.