https://llmobile.kavents.com/tags/rwkv/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/rwkv/https://llmobile.kavents.com/ticker/rwkv/Mon, 22 May 2023 15:57:41 +0200Bo Peng and 29 co-authors published RWKV on May 22, 2023, an architecture that trains in parallel like a transformer but runs as a recurrent network when it generates text. They released pretrained weights in six sizes from 169M to 14B parameters, all trained on the Pile, and describe the 14B model as by far the largest dense RNN trained to that point. A transformer keeps a cache of every token it has already processed, so both the memory it holds and the work it does for each new token grow as the context gets longer. RWKV carries a fixed-size state instead, which holds the same amount of data at token 10 and at token 10,000. The complexity table in the paper puts RWKV at O(Td) time and O(d) space for a sequence of T tokens, against O(T²d) time and O(T² + Td) space for a standard transformer, and the authors note that the arithmetic per token does not depend on context length at all. The authors measured text generation on an NVIDIA A100 with 80 GB in float32 and report cumulative time that rises in a straight line for RWKV while the transformer baselines curve upward. In their chart, RWKV-4 3B reaches about 10 seconds of cumulative GPU time after 1024 tokens, where OPT 2.7B, GPT-Neo 2.7B, BLOOM 3B and Pythia 2.8B need roughly 57 to 63 seconds. Chart: Peng et al., licensed CC BY 4.0. Cumulative time to generate 1024 tokens in float32 on an A100. On zero-shot benchmarks the paper puts RWKV close to transformers of comparable size trained on the same data. RWKV-4 at 169M scores 65.07 on PIQA where Pythia 160M scores 62.68 and GPT-Neo 125M scores 63.06, and at 14.2B it leads an interpolated GPT-level baseline on five of seven common sense tasks while trailing it on WinoGrande and ARC-Easy. The authors list the trade-off as a limitation, since funnelling everything through one fixed state limits how well the model recalls small details from very long contexts, and they report that RWKV is more sensitive to prompt wording than a transformer. Peng publishes the code as RWKV-LM under Apache 2.0, and the weights on Hugging Face carry the same license. The repository lists rwkv.cpp among community projects, a ggml-based implementation that runs 4-bit, 8-bit and float inference on CPU, while optimised iOS and Android inference sits in the README as a to-do rather than finished work. Peng states the goal in that list. RWKV is a RNN and very friendly for edge devices. Let’s make it possible to run a LLM on your phone. Source: https://arxiv.org/abs/2305.13048 Read the article: https://llmobile.kavents.com/ticker/rwkv/RWKVResearchOpen weightsMemoryOpen source