ROMA keeps a 4-bit 3B model in on-chip ROM and reports 31,800 tok/s in synthesis

Researchers at Shanghai Jiao Tong University, Peking University and Microsoft Research Asia described ROMA on March 17, 2025, an accelerator design that holds a quantised language model in on-chip read-only memory. The paper reports results from Verilog synthesis against TSMC’s 7 nm standard cell library, not from a fabricated chip and not from an FPGA prototype, so every speed, area and power figure in it is a toolchain estimate rather than a measurement on hardware. On that basis the authors report peak decoding of 31,800 tok/s for a 4-bit 3B Llama model with an empty KV cache.

The design splits storage by how often the weights change. The frozen base model, quantised to 2-bit or 4-bit, sits in ROM, which cannot be rewritten once the chip has been manufactured, while the small trainable LoRA adapter and the KV cache sit in SRAM, which can. The authors argue that a base model is stable enough for that, with updates expected over years rather than months, and that adaptation to new tasks happens in the adapter instead. What ROM buys is density, and the paper puts a ROM bit cell at roughly a third of the area of an SRAM bit cell in TSMC’s 7 nm memory compiler, enough to fit the whole quantised model on chip without external memory.

The synthesised configuration carries 1.86 GB of ROM and 304 MB of SRAM at 500 MHz, with an area of 503.7 mm² and power of 33.1 W, the area and latency coming from Synopsys Design Compiler and the power from Synopsys PrimeTime PX. It holds either a 4-bit 3B or a 2-bit 8B Llama model entirely on chip, and for the 3B model with a rank-16 adapter the authors report a time to first token of 5.6 ms at 256 input tokens and 140.2 ms at 4K, with decoding staying above 10,000 tok/s once 4K tokens are cached. They also compare against measured hardware, an Intel i5-1135G7 running llama.cpp at 6.8 tok/s and an Nvidia RTX 4090 running TensorRT-LLM at 219 tok/s on the same 4-bit 3B model, against 20,078 tok/s for the ROMA estimate at a 256-token input.

Most of the paper’s engineering goes into shrinking the ROM. A structure the authors call B-ROM stores the constant bits in the chip’s wiring instead of in transistors, which cuts the transistor count to about a quarter and the ROM area by around 40 percent against a standard ROM, and a fused cell layout places that wiring-heavy structure in the same physical area as the transistor-heavy compute logic. The paper states no tapeout, no fabrication schedule and no cost figures, and it names the fixed base model as the price of the approach, since ROM content is set at manufacture. It is posted under the arXiv perpetual non-exclusive license.