https://llmobile.kavents.com/tags/gemma/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/llama-cpp-hexagon-npu-snapdragon/https://llmobile.kavents.com/ticker/llama-cpp-hexagon-npu-snapdragon/Thu, 10 Sep 2026 11:00:00 +0200A user has reported results from llama.cpp’s Hexagon NPU backend on a OnePlus 12 with a Snapdragon 8 Gen 3. They cross-compiled llama.cpp on Ubuntu following the project’s Snapdragon documentation and ran it from Termux on the phone. The reported figures are 8 tokens per second of prompt processing and 4.5 of generation with gemma-3-12b-it-qat at Q4_0, and 20 tokens per second of prompt processing and 12.5 of generation with gemma-3-4b-it-qat at Q4_0. According to the report the speed is about the same as running on the same chip’s CPU, but the phone does not get hot. The report lists several limitations. The backend accepts only Q4_0, IQ4_NL, MXFP4, Q8_0 and F32 files, and does not support KV cache quantisation. It runs on chips as old as the Snapdragon 888, but the poster notes that only the 8 Gen 2 and newer carry the tensor module used for language models. The Hexagon NPU can address 4 GB, so a model plus KV cache beyond that has to be split across more than one NPU device: LD_LIBRARY_PATH=./lib:/vendor/lib64 \ ADSP_LIBRARY_PATH=./lib \ GGML_HEXAGON_NDEV=2 \ ./bin/llama-completion \ -m /sdcard/gguf/gemma-3-12b-it-qat-Q4_0.gguf \ -ngl 99 --device HTP0,HTP1 \ -sys 'You are a helpful AI assistant' For context the poster cites 34 INT8 TOPS and 76.8 GB/s of memory bandwidth for the 8 Gen 3 NPU, against 80 TOPS and 228 GB/s for Qualcomm’s X2 Elite Extreme and 248 TOPS and 936 GB/s for an Nvidia RTX 3090. They ask readers with a Snapdragon 8 Elite Gen 5 phone or an X2 Elite Extreme laptop to post their own numbers, since those parts are said to address more than 4 GB and would not need multiple devices. Source: https://www.reddit.com/r/LocalLLaMA/comments/1t0k6fj/running_llamacpp_on_snapdragon_hexagon_npu_seems/ Read the article: https://llmobile.kavents.com/ticker/llama-cpp-hexagon-npu-snapdragon/QualcommAndroidNPUllama.cppGemmahttps://llmobile.kavents.com/ticker/litert-lm-on-device-genai/https://llmobile.kavents.com/ticker/litert-lm-on-device-genai/Tue, 19 May 2026 18:00:00 +0200Google has detailed LiteRT-LM, the runtime it uses to run Gemma 4 on-device in Chrome, ChromeOS, the Pixel Watch and the Google AI Edge Gallery app. It sits on top of LiteRT, formerly TensorFlow Lite, and adds GenAI libraries over the XNNPACK and ML Drift kernels. Running Gemma 4 E2B without multi-token prediction, Google measures 52 decode tokens per second on the Android GPU through OpenCL, 56 on iOS through Metal, and up to 76 in the browser on WebGPU. The Android figures come from a Samsung S26 Ultra, iOS from an iPhone 17 Pro, and web from Chrome on a 2024 MacBook Pro with an M4 Max. Chart: Google. The published comparison puts LiteRT-LM prefill at 3,808 tokens per second against 292 for llama.cpp on the Android GPU, 2,878 against 1,493 for MLX on iOS, and 4,182 against 1,127 for ONNX on the web. Decode figures in the same chart are 52 against 19, 56 against 32 and 76 against 29. Multi-token prediction raises decode further. Google reports up to a 2.2x speedup, taking Gemma 4 E2B from 52 to 85 tokens per second and E4B from 21 to 47, measured on a Samsung S26 Ultra GPU. The drafter and the primary model run on the same hardware block so the shared KV cache and activations stay in local memory. Chart: Google. On memory, LiteRT-LM keeps per-layer embeddings out of memory and loads image and audio encoders only when a task needs them. Google states that the 2.58 GB Gemma 4 E2B model runs with a physical footprint of 607 MB on Apple mobile CPUs using XNNPACK weight caching. For agentic use the runtime supports Thinking Mode, which developers can stream to the interface or strip to save KV cache space: By dedicating a scratchpad for step-by-step reasoning before the model commits to an action, LiteRT-LM can significantly improve the output quality. Constrained decoding enforces JSON schemas on tool payloads, and native function calling pauses execution, returns the tool-call request to the app and resumes on the tool’s output. Session save and restore serialise KV cache state so long conversations skip the prefill phase on return. Beyond the existing Kotlin and C++ interfaces on Android, Google is adding an open-source Swift API for Apple platforms and a JavaScript API for the web through WebAssembly and WebGPU. Source: https://developers.googleblog.com/blazing-fast-on-device-genai-with-litert-lm/ Read the article: https://llmobile.kavents.com/ticker/litert-lm-on-device-genai/GoogleLiteRTGemmaBenchmarksAndroidhttps://llmobile.kavents.com/ticker/airgap/https://llmobile.kavents.com/ticker/airgap/Wed, 29 Apr 2026 08:06:11 +0200Xavier Puspus published Airgap on April 29, 2026, a React Native starter kit for branded customer support apps that have to keep answering when the network is slow, missing or deliberately blocked. Its configuration file pins a Gemma 4 E2B instruction-tuned model in Q3_K_S GGUF form, about 2.4 GB, which the app downloads on first launch and runs through llama.rn, the React Native binding for llama.cpp. The README states that Airgap searches bundled local documents first and uses a model only to phrase what that retrieval returned, and it lists Apple Foundation Models and Google’s ML Kit GenAI Prompt API as two further on-device answer providers an operator can put ahead of the downloaded file. The repository ships seven example configurations, for airline, banking, electric utility, healthcare, insurance, telecom and water utility support, each with its own branding, prompts and JSON knowledge files, and the downloaded-model path targets iOS 15.1 and newer and Android API 24 and newer. Airgap is published under the MIT license. It is an early single-author project, with 54 commits and no tagged release, and almost all of the code arrived in one squashed initial commit followed by a burst of work in August 2026. The README lists physical-device evaluation of the Apple and Android system providers as an open release gate and labels the published demo recordings as Android emulator and iOS simulator footage rather than evidence of either system model running on a phone. Source: https://github.com/xmpuspus/airgap Read the article: https://llmobile.kavents.com/ticker/airgap/Open sourceAndroidiOSGemmallama.cpphttps://llmobile.kavents.com/ticker/gemma-4-edge-agent-skills/https://llmobile.kavents.com/ticker/gemma-4-edge-agent-skills/Thu, 02 Apr 2026 18:00:00 +0200Google DeepMind has launched Gemma 4 under the Apache 2.0 licence, and the Google AI Edge team has described how the E2B and E4B variants run on edge hardware. The models support more than 140 languages and a 128K context window. Gemma 4 enables multi-step planning, autonomous action, offline code generation, and even audio-visual processing, all without specialized fine-tuning. Image: Google. The Google AI Edge Gallery app on iOS and Android gains Agent Skills, which Google calls one of the first applications to run multi-step, autonomous agentic workflows entirely on-device. The examples given are querying Wikipedia, turning speech input into charts and flashcards, pairing photos with generated music, and building a small app that plays animal calls through conversation alone. For in-app deployment Google points to LiteRT-LM. It reports that Gemma 4 E2B runs in under 1.5 GB of memory on some devices using 2-bit and 4-bit weights with memory-mapped per-layer embeddings, and that GPU optimisations process 4,000 input tokens across two skills in under 3 seconds. Constrained decoding enforces structured output, and dynamic context lengths let a single model span CPUs and GPUs. On smaller hardware, LiteRT-LM reaches 133 prefill and 7.6 decode tokens per second on a Raspberry Pi 5 CPU, and 3,700 prefill and 31 decode tokens per second on the NPU of a Qualcomm Dragonwing IQ8, the processor behind the Arduino VENTUNO Q. Gemma 4 is available with CPU and GPU support on Android and iOS, system-wide on Android through the new AICore Developer Preview, on Windows, Linux and macOS via Metal, and in the browser through WebGPU. Google also released a Python package and a litert-lm CLI for Linux, macOS and Raspberry Pi, with tool calling included. Source: https://developers.googleblog.com/bring-state-of-the-art-agentic-skills-to-the-edge-with-gemma-4/ Read the article: https://llmobile.kavents.com/ticker/gemma-4-edge-agent-skills/GoogleGemmaAgentsOpen weightsLiteRThttps://llmobile.kavents.com/ticker/gemma-3n/https://llmobile.kavents.com/ticker/gemma-3n/Tue, 20 May 2025 19:00:00 +0200Google previewed Gemma 3n on May 20, 2025, a model built for phones, tablets and laptops. Its 5B and 8B versions run with dynamic memory footprints of 2 GB and 3 GB, which Google says matches what 2B and 4B models normally need. The saving comes from Per-Layer Embeddings, a Google DeepMind technique that keeps embedding parameters off the accelerator. For E2B that cuts what has to be loaded into accelerator memory from 5.44B parameters to 1.91B, with the 2.55B embedding parameters cached to fast storage instead. A MatFormer setup nests a smaller but fully functional 2B model inside the 4B one, so developers can trade quality against footprint. Google reports the model starting to respond about 1.5 times faster on mobile than Gemma 3 4B. Gemma 3n takes audio, text, images and video as input, including interleaved inputs across modalities, and handles speech recognition and translation. Google states that the same architecture powers the next generation of Gemini Nano, which was to reach Android and Chrome later in 2025. Qualcomm Technologies, MediaTek and Samsung System LSI worked on the optimisation. Diagram: Google. ▶Announcing Gemma 3n Preview: Powerful, Efficient, Mobile-First AILoading connects your browser to www.youtube-nocookie.com, which may process your IP address and use cookies. Load contentOpen externallyVideo: Google for Developers. Update, June 26, 2025. Google released Gemma 3n as E2B and E4B, naming them for 2B and 4B effective parameters out of 5B and 8B raw ones. Google reports E4B scoring 1303 on LMArena and calls it the first model under 10 billion parameters to pass 1300, ahead of Llama 4 Maverick 17B 128E at 1292 and GPT 4.1-nano at 1288. The release adds a MobileNet-V5-300M vision encoder, which Google says handles up to 60 frames per second on a Pixel, and KV cache sharing that doubles prefill performance. Text covers 140 languages, multimodal understanding 35. Weights are on Hugging Face and Kaggle, with support in Ollama, llama.cpp, MLX, LiteRT and the Google AI Edge Gallery. Chart: Google. Source: https://developers.googleblog.com/en/introducing-gemma-3n/ Read the article: https://llmobile.kavents.com/ticker/gemma-3n/GoogleGemmaGemini NanoAndroidOpen weightshttps://llmobile.kavents.com/ticker/gemma-3/https://llmobile.kavents.com/ticker/gemma-3/Wed, 12 Mar 2025 17:00:00 +0100Google released Gemma 3 on March 12, 2025 in four sizes, 1B, 4B, 12B and 27B, and describes them as designed to run fast directly on devices, from phones and laptops to workstations. The 1B is a new size for the line, which started at 2B with Gemma and stayed there through Gemma 2. The technical report puts the 1B at 2.0 GB of weights in bfloat16 and at 0.5 GB as a quantised int4 checkpoint, and the 4B at 8.0 GB and 2.6 GB. Context length is not the same across the four. The report gives 128K tokens for the 4B, 12B and 27B and 32K for the 1B, with the three larger models pre-trained at 32K and stretched to 128K at the end of pre-training by rescaling their rotary position embeddings. To stop the key-value cache from swallowing memory at long context, Gemma 3 places five local attention layers between each global one and holds the local layers to a 1024-token window, where Gemma 2 alternated one for one. Google reports that in an ablation on a 2B model a global-attention-only configuration carries a 60 percent memory overhead from the cache at a 32K prefill, and that interleaving with a 1024-token window brings that under 15 percent. Multimodality starts at 4B. The 4B, 12B and 27B share one 400M-parameter SigLIP vision encoder, frozen during training, which takes square images resized to 896 by 896 pixels and condenses each one into 256 tokens for the language model to read. An inference-time step Google calls Pan and Scan cuts non-square or high-resolution images into equal crops and resizes each to the encoder’s resolution, which the report measures as raising the 4B from 72.8 to 81.0 on DocVQA and from 44.1 to 57.0 on InfoVQA in a 4-shot evaluation of a pre-trained checkpoint. The 1B has no vision encoder and takes text only. Google published quantisation-aware trained checkpoints next to the raw weights, made by fine-tuning each model for about 5,000 steps against the probabilities of the unquantised model, so the weights are trained to tolerate the lower precision rather than rounded down afterwards. The three formats are per-channel int4, per-block int4 and switched fp8, which Google picked for what open source inference engines such as llama.cpp already support. The report’s footprint table counts weights alone and weights plus a key-value cache for a 32,768-token sequence, and on the second measure the 1B needs 1.4 GB at per-channel int4 against 2.9 GB in bfloat16, while the 4B needs 7.3 GB against 12.7 GB. Google's own footprint figures for the raw and quantised checkpoints, with and without a key-value cache at 32,768 tokens. Table: Google DeepMind, Gemma 3 technical report, CC BY 4.0. Google reports its own zero-shot figures for the instruction-tuned models and puts Gemma 3 4B at 43.6 on MMLU-Pro and 75.6 on MATH, against 15.6 and 27.2 for Gemma 2 2B and 56.9 and 55.6 for Gemma 2 27B, with the 1B at 14.7 and 48.0. The 4B also scores 48.8 on the MMMU validation set, a benchmark the text-only sizes do not run, and the report calls the instruction-tuned 4B competitive with the instruction-tuned Gemma 2 27B. Google put every size on Hugging Face, Kaggle and Ollama as a pre-trained and an instruction-tuned checkpoint under the Gemma Terms of Use, and names LiteRT, Gemma.cpp, llama.cpp and Google AI Edge among the runtimes that take them. The same release added ShieldGemma 2, a 4B image safety classifier built on Gemma 3 that labels images for dangerous content, sexually explicit material and violence. ▶What's new in Gemma 3?Loading connects your browser to www.youtube-nocookie.com, which may process your IP address and use cookies. Load contentOpen externallyVideo: Google for Developers. Source: https://blog.google/technology/developers/gemma-3/ Read the article: https://llmobile.kavents.com/ticker/gemma-3/GoogleGemmaQuantisationOpen weightsBenchmarkshttps://llmobile.kavents.com/ticker/gemma-2/https://llmobile.kavents.com/ticker/gemma-2/Wed, 31 Jul 2024 17:00:00 +0200Google released Gemma 2 2B on July 31, 2024, an open model of 2.6B parameters that Google says produces outsized results by learning from larger models through distillation. Google reports it at 1126 Elo on the LMSYS Chatbot Arena leaderboard, captured on July 30, 2024, ahead of Mixtral 8x7B Instruct at 1114, GPT 3.5 Turbo 0314 at 1106 and Llama 2 70B chat at 1093, and states that the model surpasses all GPT-3.5 models there. Chart: Google, from the announcement. Scores are Google's own reading of the leaderboard on July 30, 2024. The technical report describes training the 2B size on 2 trillion tokens with knowledge distillation, where a bigger teacher model supplies a probability for every token in context and the small student is trained to match those probabilities rather than to predict the next token. In an ablation the authors ran, a 2B model trained over 500B tokens distilled from a 7B teacher averaged 67.7 across three benchmarks, against 60.3 for the same model trained from scratch. The released model holds 2.02B non-embedding parameters plus 590M embedding parameters, which come from the 256k-entry Gemini vocabulary, and it interleaves local sliding-window attention with global attention over an 8192-token context. Google DeepMind trained it on 512 TPUv5e chips. Google names edge devices and laptops as deployment targets next to cloud serving on Vertex AI and Google Kubernetes Engine, and says the model is small enough for the free tier of T4 GPUs in Google Colab. For running it locally Google points to Ollama and to Gemma.cpp, its own lightweight C++ inference engine, and says support in MediaPipe, the on-device inference stack for Android, iOS and the web, is coming. Google adds that NVIDIA optimised the model with the TensorRT-LLM library for GeForce RTX cards and for Jetson modules at the edge. Google put the weights on Kaggle, Hugging Face and Vertex AI Model Garden under the Gemma terms, which it calls commercially friendly for research and commercial applications, and made the model available to try in Google AI Studio. The same announcement added ShieldGemma, a set of safety classifiers built on Gemma 2 that filter hate speech, harassment, sexually explicit and dangerous content, and Gemma Scope, over 400 sparse autoencoders for inspecting the inner workings of Gemma 2 2B and 9B. Source: https://developers.googleblog.com/en/smaller-safer-more-transparent-advancing-responsible-ai-with-gemma/ Read the article: https://llmobile.kavents.com/ticker/gemma-2/GoogleGemmaDistillationOpen weightsBenchmarkshttps://llmobile.kavents.com/ticker/octopus-v2/https://llmobile.kavents.com/ticker/octopus-v2/Tue, 02 Apr 2024 11:01:00 +0200Wei Chen and Zhiyuan Li published Octopus v2 on April 2, 2024, a 2B function-calling model fine-tuned from Google’s Gemma 2B that turns a spoken or typed request into an Android API call. On their own evaluation set built from 20 Android APIs, the authors report 99.524% accuracy and 0.38 seconds per call for Octopus v2, against 98.571% and 1.02 seconds for GPT-4. Both authors give Stanford University as their affiliation on the paper and publish the weights under the name Nexa AI. The distinguishing idea is what the authors call a functional token. Each of the 20 APIs gets its own special token added to the tokenizer and to the model’s output head, so picking a function becomes a single choice among 20 token candidates instead of writing the function name out as text and hoping every piece of it lands correctly. Since the model learned during fine-tuning what each of those tokens stands for, the function descriptions no longer have to sit in the prompt, which the paper puts at over 95% less context. Retrieval-augmented function calling processes roughly 1,000 tokens per call in their comparison, while Octopus v2 sees the query alone, typically under 30 tokens. Shorter context is also where the speed comes from. The authors measure Llama-7B with retrieval-augmented function calling at 68.095% accuracy and 13.46 seconds per call on a single NVIDIA A100, and Octopus v2 at 0.38 seconds under the same settings on the same A100, a 35-fold difference. The phone figure is a separate measurement. With the model quantised and the state for the fixed prompt prefix precomputed, they report a function call finishing in 1.1 to 1.7 seconds for queries of 20 to 30 tokens on what the paper calls a standard Android phone, without naming the device, and they estimate 37 times more calls per iPhone battery charge than a 7B model with retrieval. The authors name limits of their own result. Training with LoRA instead of the full model drops accuracy to 99.048%, and cutting the training set from 1,000 to 100 generated examples per API drops it to 98.095%, which they still put forward as the low end of a usable range. The set of callable functions is fixed at training time, since every API has to exist as a token before the model can emit it, and they price the data generation at 0.0224 USD per 1,000 examples per API. They also note that the GPT-3.5 and GPT-4 latencies were taken through OpenAI’s API on March 18, 2024, and may reflect API traffic or hardware allocation at that moment. Nexa AI released the weights on Hugging Face under a non-commercial CC BY-NC 4.0 license, on top of Google’s Gemma 2B. The model card said at the time that the team was still preparing the code for an open-source release. Source: https://arxiv.org/abs/2404.01744 Read the article: https://llmobile.kavents.com/ticker/octopus-v2/Nexa AIAgentsAndroidGemmaResearchhttps://llmobile.kavents.com/ticker/octopus-function-calling/https://llmobile.kavents.com/ticker/octopus-function-calling/Tue, 02 Apr 2024 03:29:28 +0200Wei Chen, Zhiyuan Li and Mingyuan Ma of Stanford University and Harvard University posted Octopus on April 2, 2024, four open models fine-tuned to pick the right software API for a request and fill in its arguments. On a benchmark the authors built themselves, they report their fine-tuned Gemma 2B answering 93 percent of calls correctly against 96 percent for GPT-4 and 50 percent for GPT-3.5, with the CodeLlama 7B and Gemma 7B versions reaching 97 percent once an output filter is switched on. The title says on-device, but nothing in the paper runs on a device. The authors fine-tuned the models on A100 80GB GPUs and queried GPT-4 and GPT-3.5 through OpenAI’s API, so every number was measured server-side. The device claim is a single sentence stating that models of this size can already be deployed on mobile, pointing at MLC LLM, and the paper names no phone, no chip and no latency, memory or throughput figure. The training set starts from roughly 30,000 of the most used APIs on RapidAPI Hub, which GPT-4 cut to about 20,000 by dropping entries with missing arguments or descriptions that did not match their parameters. The authors rewrote each API as a Python function signature with a docstring, had GPT-4 write five answerable queries per API and an equal share of unanswerable ones, and let GPT-4 verify the result, ending at about 150,000 examples as described in the paper. Vector search over the function descriptions pulled in near-identical functions as distractors, and training worked through them from easy to hard. Base models were CodeLlama 7B, Gemma 2B and 7B and Stable Code 3B, all tuned with LoRA at rank 16 and 8-bit quantisation, using 90 GPU hours for CodeLlama 7B and Gemma 7B, 60 for Stable Code 3B and 30 for Gemma 2B. The output filter the authors call a conditional mask blocks, at each generated token, anything that would break the expected format, so a function name has to come from the known list and an argument has to match its declared type. The authors state that it cannot be applied to GPT-4 or GPT-3.5 because the OpenAI API does not return the underlying token scores, and that GPT-3.5 was not counted wrong for formatting mistakes. The paper states that the training dataset and the fine-tuned models will be open sourced soon, and names no repository or date. The preprint went up under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license. Source: https://arxiv.org/abs/2404.01549 Read the article: https://llmobile.kavents.com/ticker/octopus-function-calling/Nexa AIResearchAgentsGemmaBenchmarkshttps://llmobile.kavents.com/ticker/gemma/https://llmobile.kavents.com/ticker/gemma/Wed, 21 Feb 2024 18:00:00 +0100Google released Gemma on February 21, 2024, the first two models in the line, at 2B and 7B parameters and in a pretrained and an instruction-tuned checkpoint each. The technical report presents the 7B as a model for deployment on GPU and TPU and the 2B as one for CPU and on-device applications. Google says Gemma was built from the same research and technology used to create Gemini and shares technical and infrastructure components with it. Both sizes are decoder-only transformers trained on a context length of 8192 tokens. The 2B has 18 layers, a model dimension of 2048 and a single key-value head, since Google’s ablations found multi-query attention works well at small scale, while the 7B has 28 layers and keeps standard multi-head attention across 16 heads. Both inherit Gemini’s 256k-entry vocabulary, which puts 524M of the 2B’s parameters into embeddings and leaves 1.98B elsewhere. Google reports training the 2B on 3T tokens and the 7B on 6T, mostly English web documents, mathematics and code. Google reports Gemma 7B at 64.3 on MMLU 5-shot against 54.8 for Llama-2 13B, 46.4 on GSM8K against 28.7, and 32.3 on HumanEval against 18.3, for an average of 56.9 across 18 academic benchmarks. Google puts the 2B at 42.3 on MMLU and 45.0 on average, ahead of Llama-2 7B on the mathematics and coding tasks and behind it overall. Google states that Gemma outperforms similarly sized open models on 11 of the 18 text-based tasks, and notes it could not rerun the Llama-2 evaluations itself because of that model’s licensing, so it cites Meta’s published figures. Google's own figures for Gemma 7B against Llama-2. Chart: Google. The weights went up on Kaggle and Hugging Face, with the models also runnable from Colab and Vertex AI, and Google provided toolchains for inference and supervised fine-tuning across JAX, PyTorch and TensorFlow through native Keras 3.0. Hugging Face added Gemma to Transformers 4.38 on announcement day. llama.cpp merged Gemma support the same day, within half an hour of the pull request opening, which is what brought the 2B into quantised local runs on consumer hardware. Google states that the models run across laptop, desktop, IoT, mobile and cloud. The weights are open but the licence is not a standard open source one. Google publishes them under its own Gemma Terms of Use, which allow use, modification and redistribution provided that downstream recipients get the same terms and the separate Prohibited Use Policy, and which reserve Google’s right to restrict uses it considers non-compliant. Google says the terms permit responsible commercial usage and distribution for all organisations regardless of size. Alongside the models Google shipped a Responsible Generative AI Toolkit with a safety classification method, a tool for debugging model behaviour and written guidance for model builders. Source: https://blog.google/technology/developers/gemma-open-models/ Read the article: https://llmobile.kavents.com/ticker/gemma/GoogleGemmaOpen weightsllama.cppBenchmarks