https://llmobile.kavents.com/tags/stability-ai/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/stable-lm-2-zephyr/https://llmobile.kavents.com/ticker/stable-lm-2-zephyr/Tue, 27 Feb 2024 20:00:00 +0100Stability AI published the Stable LM 2 1.6B technical report on February 27, 2024, describing a decoder-only model of 1.6B parameters trained from scratch on roughly 2 trillion tokens of public data. The mix is multilingual by design, with web text in German, Spanish, French, Italian, Dutch and Portuguese alongside English, and the report states the finished model handles all seven. The model has 24 layers and a context length of 4096 tokens, which the authors list as one of the things they want to extend. In the report’s multilingual table, which averages translated versions of four English benchmarks and a next-word prediction set across those seven languages, the base model scores 40.5, above Google’s Gemma 2B at 39.8 with 2.5B parameters and Microsoft’s Phi-2 at 34.6 with 2.7B. The chat variant reaches 41.5 and Stability’s own 3B model 41.7. The authors state they used no multilingual data during fine-tuning, so the language coverage comes out of pre-training alone. On the six English tasks of the Hugging Face Open LLM Leaderboard, the same report’s table puts the base model at an average of 45.3, behind Alibaba’s Qwen 1.5 1.8B at 46.6 and Gemma 2B at 46.5. The instruction-tuned StableLM 2 Zephyr 1.6B averages 49.7, the highest of the sub-2B entries in that table, ahead of Phi-1.5 at 47.7 and behind the 2.7B Phi-2 at 61.3. The report’s abstract calls Stable LM 2 1.6B the state-of-the-art open model under 2B parameters at the time of publication. Stability built the Zephyr variant in three stages. It fine-tuned the base model on seven public instruction datasets, 826,938 samples after dropping conversations longer than eight turns, then aligned the result with direct preference optimisation, which trains on pairs of preferred and rejected answers, using UltraFeedback and Intel’s Orca Pairs and borrowing most hyperparameters from the recipe behind Hugging Face’s Zephyr 7B. A third stage taught the model who made it and what a language model cannot do, using 10,000 opening messages the base model wrote for itself. The variant scores 5.42 on MT-Bench, where a larger model judges answers to open-ended questions, above Qwen 1.5 1.8B Chat at 5.29 and Gemma 2B Instruct at 5.19 and below Stability’s own StableLM Zephyr 3B at 6.64. Chart: Bellagente et al., the authors' own figures. Stability put both sets of weights on Hugging Face along with quantised builds for llama.cpp, Apple’s MLX and Intel’s OpenVINO. The report measures 127 tok/s at 11 W for the 4-bit MLX build on an M2 Mac mini with 8 GB, against 71 tok/s at 6 W in FP16, and 99 tok/s for a 4-bit GGUF build on a 2023 MacBook Pro with 16 GB, while stating these figures are meant as a practical reference rather than rigorous benchmarking. Neither release is open source. The report says the model ships under a non-commercial licence, the Zephyr model card names the Stability AI Non-Commercial Research Community License and points commercial users to the company, and the base model card now carries the Stability AI Community License with commercial use running through a Stability membership. Source: https://arxiv.org/abs/2402.17834 Read the article: https://llmobile.kavents.com/ticker/stable-lm-2-zephyr/Stability AIResearchBenchmarksOpen weightsApple Siliconhttps://llmobile.kavents.com/ticker/stable-lm-zephyr/https://llmobile.kavents.com/ticker/stable-lm-zephyr/Thu, 07 Dec 2023 17:00:00 +0100Stability AI released StableLM Zephyr 3B on December 7, 2023, a chat model with 3 billion parameters that the company aims at edge devices and hardware without the compute of dedicated high-end systems. Stability calls the model 60% smaller than 7B models and reports an MT-Bench score of 6.64 and an AlpacaEval win rate of 76.00%. Both numbers are Stability’s own, measured on two evaluations the company links from the announcement. MT-Bench scores answers to open-ended questions with a larger language model acting as judge, and AlpacaEval measures how often a judge prefers a model’s reply over a reference answer. In the table Stability published with the model card, the 3B model sits above Falcon-Instruct at 40B parameters, which scored 5.17 on MT-Bench, and MPT-Chat at 7B with 5.42, and below Zephyr-7b-beta at 7.34 and Llama2-Chat at 70B with 6.86. Table: Stability AI, the company's own benchmark figures. The model extends Stability’s earlier StableLM 3B-4e1t and borrows its training pipeline from Hugging Face’s Zephyr 7B, according to Stability. Stability first ran supervised fine-tuning on the UltraChat, MetaMathQA, Evol Wizard and Capybara instruction datasets, then aligned the result with direct preference optimisation, which trains a model on pairs of preferred and rejected answers instead of on correct answers alone. That second stage used UltraFeedback from the OpenBMB research group, a set of 64,000 prompts with model responses. Stability put the weights on Hugging Face and linked an example notebook for tuning the model’s speed. The release is not open source. Stability states that it ships under a non-commercial community licence permitting non-commercial use, and tells anyone who wants the model in a commercial product to contact the company for separate terms. Source: https://stability.ai/news/stablelm-zephyr-3b-stability-llm Read the article: https://llmobile.kavents.com/ticker/stable-lm-zephyr/Stability AIOpen weightsBenchmarks