Qualcomm argues for hybrid AI, citing 10 times the cost per generative AI search query
Qualcomm published a whitepaper, The future of AI is hybrid, in May 2023 arguing that generative AI inference has to be distributed between the cloud and edge devices rather than run in data centres alone. The company’s case rests on cost. It cites a Reuters report estimating that a generative AI search costs 10 times more per query than traditional search, and a Morgan Stanley note from February 2023 saying that with more than 10 billion search queries a day, even a small share moving to language models could add multiple billions of dollars a year.
On what can run locally, Qualcomm writes that models of 1 to 10 billion parameters, assuming INT4 weights, cover capabilities including text-to-image, dialog, image understanding and video understanding. It states that models with more than 1 billion parameters, such as Stable Diffusion, already run on phones with performance and accuracy similar to the cloud equivalent, and that models of 10 billion parameters or more are slated to run on devices in the near future. The chart in the paper puts the phone and laptop reach at about 10 billion parameters for 2023 and extends it further for 2024, while the cloud range spans the whole axis to 1000 billion.
The paper sets out three ways of splitting a workload. In the device-centric option the device runs most of the inference and an on-device neural network or rules-based arbiter decides when a query needs the cloud, with Qualcomm expecting laptops to run models of up to tens of billions of parameters. In the device-sensing option, speech recognition such as Whisper, computer vision and text-to-speech run on the phone while a cloud model does the language work, which the company argues cuts both compute and bandwidth.
The third option has device and cloud work on the same response through speculative decoding, a technique published by Google researchers in 2022. Qualcomm’s example has a draft model 7 to 10 times smaller than the target model generate four tokens on the device, which the cloud then checks by running the target model once across the four candidates, since a model like GPT-3 has to read all 175 billion parameters from memory to produce a single token. The company says its own early experiments and published results show 2 to 3 of the 4 draft tokens being accepted on average.
The only on-device demonstration the paper cites is Qualcomm’s own Stable Diffusion run on an Android phone from February 2023. Its other device claims are projections, including text-to-image models generating 3D textures on smartphones within a year and a 13 billion parameter LLaMA model reaching edge devices to voice avatars in XR. On energy the document argues only that edge devices run generative AI at a fraction of the cloud’s consumption once data transport is counted, without giving a figure. The whitepaper is Part I of two, announced in a Qualcomm blog post by Ziad Asghar and Jilei Hou, with Part II covering the company’s own products.