Amazon survey puts some small models at 10 to 100 times their parameter count
Shreyas Subramanian, Vikram Elango and Mecit Gungor of Amazon surveyed about 160 papers on small language models on January 3, 2025, covering general-purpose models under 8B parameters plus a few exceptions up to 13B. Their central claim is that some of these models reach the performance of models 10 to 100 times their parameter count, a ratio the authors call effective size and anchor to PaLM at 540B parameters. In their table, Phi-1 at 1.3B parameters is credited with an effective size of 175B on code benchmarks and Phi-2 at 2.7B with 70B.
The survey is clear about where that stops. Its appendix prints completions from 14 models for the prompt “The diagonals of a rectangle”, where a 33M TinyStories model answers “and they were so beautiful” and a 125M GPT Neo model loops on its own words, while TinyLlama at 1.1B and Phi 1.5 at 1.3B give the correct definition. The authors conclude that several billion parameters are the minimum for usable general-purpose responses, and they caution that every effective size they report comes from each paper’s best reported benchmark, so results on other benchmarks can be much lower. The clearest wins they list are task-specific and stay inside one domain, with WizardMath at 7B beating Llama 2 70B on maths benchmarks and Code Llama at 7B beating it on coding benchmarks.
On what closes the gap, the survey puts training data first. It credits Phi-1 with 50.6 percent on HumanEval at 1.3B parameters trained on textbook-quality synthetic data, and TinyGSM, where a 1.3B generator paired with a 1.3B verifier scored 81.5 on the GSM8K maths set against 77.4 for the GPT-3.5 model that wrote its training data. Distillation covers the training side, with Orca 2 at 13B leading Llama 2 70B by an average of 3.23 points after learning step-by-step explanations from a teacher model rather than just its answers. After training, the survey groups quantisation and pruning, citing SmoothQuant for 8-bit weights and activations at up to 1.56 times the speed and half the memory, and Sheared-LLaMA for cutting Llama 2 7B down to 1.3B and 2.7B using 3 percent of the compute that training those sizes from scratch would need.
The authors name resource-constrained deployment as the main motivation for the whole field, listing mobile devices, edge computing and regions with limited connectivity, and point at SmolLM at 135M, 360M and 1.7B parameters as a family built to run locally on phones and laptops. They note that extreme quantisation and sparse training already put Llama inference on CPUs within reach, and that 8-bit Adam training of a 7B model fits on a single 24 GB Nvidia RTX 4090. The paper is on arXiv under a CC BY-NC-ND 4.0 license, and its authors maintain a leaderboard of small models on Hugging Face Spaces.