https://llmobile.kavents.com/tags/agents/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.kavents.com/ticker/rikkahub-agent-local-llm/https://llmobile.kavents.com/ticker/rikkahub-agent-local-llm/Sat, 08 Aug 2026 00:00:00 +0200Adam Conway at XDA has tested RikkaHub Agent, a fork by ExTV of the open-source Android LLM client RikkaHub, which adds an agent layer with more than 80 device tools. The phone runs the agent loop and executes the tools; the model itself runs elsewhere on the local network over an OpenAI-compatible endpoint. The test device was an Oppo Find N5 with a Snapdragon 8 Elite and 16 GB of RAM. Inference ran on a Radeon RX 7900 XTX with Qwen 3.6 27B at Q4_K_M, roughly 16 GB, at 40 to 50 tokens per second. A second configuration used Qwen 3.6 35B on a Lenovo ThinkStation PGX with Nvidia’s GB10 Grace Blackwell. Screenshot: XDA. The tool set covers tapping, swiping, scrolling and typing on screen, reading notifications, sending SMS, file management, browser control, SSH, scheduled jobs, a JavaScript runtime and Telegram. All tools are off by default and enabled per assistant, actions with side effects require approval, and a deterministic blocklist rejects destructive commands. The default context is 7,000 input tokens, rising to 32,000 with every tool enabled. Asked whether whisper.cpp was installed, the agent determined that neither the command-line tool nor a model was present, requested permission, then installed the dependencies, cloned the repository and compiled it inside Termux. The build took seven minutes, and the agent chose the 75 MB small model on speed and accuracy grounds. It then transcribed a Telegram voice message, installing ffmpeg, converting the OGG/Opus file to 16 kHz mono WAV and returning a transcript in about two seconds. Running the model on the phone itself was slower. With Gemma 4 E2B, a 2.6 GB build accelerated by LiteRT-LM, Conway reports chat worked but agentic tasks were painful, and tool calls were emitted as plain text until an update fixed the behaviour. Frontier intelligence is never what I’d want from a phone agent, which is why a local LLM works perfectly. Source: https://www.xda-developers.com/turned-phone-ai-agent-powered-local-llm-installed-software/ Read the article: https://llmobile.kavents.com/ticker/rikkahub-agent-local-llm/AndroidAgentsQwenOpen sourceSpeechhttps://llmobile.kavents.com/ticker/lfm2-5-2-6b/https://llmobile.kavents.com/ticker/lfm2-5-2-6b/Tue, 04 Aug 2026 17:00:00 +0200Liquid AI has released LFM2.5-2.6B, a 2.6-billion-parameter model built on its LFM2 architecture for edge deployment. The company reports 30 tokens per second on a phone, with CPU inference staying under 2.5 GB of memory. Its published chart identifies the device as a Qualcomm Snapdragon Galaxy (SM-S948U1) and puts memory use at 2,448 MB at Q4_K_M quantisation with a 4K-token input context. Chart: Liquid AI. On other CPUs Liquid AI cites 220 tokens per second on an M5 Max and 113 tokens per second on a Ryzen AI Max+ 395. The model was pre-trained on roughly 34 trillion tokens, uses a 128K vocabulary, and its mid-training included a dedicated 128K context-extension phase. Liquid AI positions the model for agentic use, describing planning, tool calling and multi-step tasks, and states that it is competitive with, and often outperforms, models nearly four times its size on instruction-following and tool use, while larger models keep an advantage in coding. Weights are on Hugging Face in BF16, GGUF for llama.cpp, and ONNX. Source: https://www.liquid.ai/blog/lfm2-5-2-6b Read the article: https://llmobile.kavents.com/ticker/lfm2-5-2-6b/Liquid AIOpen weightsLFMBenchmarksAgentshttps://llmobile.kavents.com/ticker/qualcomm-amon-agents-new-app/https://llmobile.kavents.com/ticker/qualcomm-amon-agents-new-app/Tue, 16 Jun 2026 07:08:00 +0200Qualcomm chief executive Cristiano Amon says the company is working on more than 40 designs of new AI devices, CNBC reports from an interview on its “The Tech Download” podcast. He listed jewellery, earbuds with cameras, pins and watches among the form factors. Amon described a shift in what devices are organised around: The phone is around the agent. The new classes of devices … are going to be around the agent as well. And the agent will be the one that will understand human intentions and will do things for you, so there is a shift in what the center of gravity is. He said phones will not disappear. On software, he told CNBC that apps are “not dead” but “are going to change”, adding: “Those agents are going to be the new app.” His example was an agent that retrieves banking transaction details without the user navigating the app. On smart glasses, Amon put current shipments in the “order of multiple tens of millions” a year and said that in “a couple of years” the category could reach the “order of hundreds of millions of glasses and could become as big as smartphones”. CNBC cites Counterpoint Research figures of 1.26 billion smartphones shipped in 2025, about 3 percent above the prior year. Amon linked the entry of AI companies into hardware, such as OpenAI’s purchase of Jony Ive’s startup io, to both agent endpoints and training data, saying the devices will gather data on a scale that is “exponentially larger” than the data used to train current models. He said Qualcomm’s chip roadmap is being reworked in response: “An entire roadmap, because I believe none of the devices we have today are prepared for the future.” The full interview is available as a video on CNBC. Source: https://www.cnbc.com/2026/06/16/qualcomm-ceo-ai-devices-agents.html Read the article: https://llmobile.kavents.com/ticker/qualcomm-amon-agents-new-app/QualcommAgentsWearablesSmart glasseshttps://llmobile.kavents.com/ticker/caped-privacy-defence/https://llmobile.kavents.com/ticker/caped-privacy-defence/Wed, 10 Jun 2026 22:48:00 +0200Researchers at the Chinese University of Hong Kong, Southeast University and Shandong University published CAPED on June 10, 2026, a filter that blacks out parts of a phone’s screenshots before they reach a cloud-hosted GUI agent. Across a 28-task privacy suite the authors built themselves, weighted seeded leakage falls from 0.766 with raw screenshots to 0.268 with the filter in front, counted only on runs where the agent still finished its task. A multimodal model plays the attacker and scores what it can still infer from the whole screenshot sequence, not from single frames. A screenshot-based agent operates a phone the way a person does, by looking at the screen, so every image it uploads carries whatever happens to sit next to the thing the user asked for. The authors call this incidental visual privacy exposure and name contacts, message previews, non-target photos, filenames, recommendation feeds and health widgets as content the remote model receives although the user never mentioned it. In one of their test shops, a request to buy sunglasses puts prenatal vitamins, a baby bottle set and a maternity dress in front of the model, because the shop’s recommendation rows share the page with the product. Stripping names and numbers out of the text does not close that gap, and the authors measure text-only anonymisation at 0.752 against 0.766 for no protection at all. Figure: Shen et al., licensed CC BY 4.0. The mock SunShop task asks the agent to buy sunglasses, so the personalised rows around it count as incidental exposure. CAPED decides per interface element rather than per screen, according to the paper. It first sorts the current screen into one of three postures, so galleries, maps, files and financial views show their text but hide their images, while messaging, contacts, feeds and health dashboards hide both by default. The user’s instruction is turned into a short list of what the agent is allowed to need, and a hidden element is released only when it matches that list, checked either by text matching or by asking a vision model whether a cropped thumbnail really shows the requested subject. Hidden areas keep their place in the layout under a solid overlay with an eye icon, and names in exposed text are replaced by placeholders the remote agent can still follow. The defence is meant to run on the phone, and the authors argue the instruction has to be interpreted locally too, since it can name people, relationships, locations and intentions before any screen policy applies. The prototype they measured does not run on a phone. They state that CAPED’s components ran on the evaluation host with an evaluation-side service for the heavier models, and that the timings should be read as live-loop prototype latency rather than physical-phone latency, energy or thermal behaviour. Over 1,394 measured steps the privacy work added 2.325 seconds per agent step, 22.6 percent of a 10.835-second step, most of it the detector at 1.811 seconds, plus 5.716 seconds once per task to extract the requirements. The protection also costs task success. On AndroidWorld’s 116 ordinary tasks, GUI-Owl-7B completed 77 of them (66.4%) unprotected and 64 (55.2%) behind the full filter, which the authors call a prototype-level utility cost. They name further limits themselves, writing that screens their rule base does not cover fall back to the most permissive posture, which they describe as not privacy-conservative, and that open-ended requests such as finding the latest receipt widen exposure and account for much of the leakage that remains. On the connected Gmail, Drive and Photos tasks the filter moves leakage from 0.402 to 0.347 only, the interactive unmasking path was never tested with real users, and the authors write that they plan to release the CAPED artifact in a future public version. Source: https://arxiv.org/abs/2606.12666 Read the article: https://llmobile.kavents.com/ticker/caped-privacy-defence/AgentsAndroidPrivacyResearchBenchmarkshttps://llmobile.kavents.com/ticker/gemma-4-edge-agent-skills/https://llmobile.kavents.com/ticker/gemma-4-edge-agent-skills/Thu, 02 Apr 2026 18:00:00 +0200Google DeepMind has launched Gemma 4 under the Apache 2.0 licence, and the Google AI Edge team has described how the E2B and E4B variants run on edge hardware. The models support more than 140 languages and a 128K context window. Gemma 4 enables multi-step planning, autonomous action, offline code generation, and even audio-visual processing, all without specialized fine-tuning. Image: Google. The Google AI Edge Gallery app on iOS and Android gains Agent Skills, which Google calls one of the first applications to run multi-step, autonomous agentic workflows entirely on-device. The examples given are querying Wikipedia, turning speech input into charts and flashcards, pairing photos with generated music, and building a small app that plays animal calls through conversation alone. For in-app deployment Google points to LiteRT-LM. It reports that Gemma 4 E2B runs in under 1.5 GB of memory on some devices using 2-bit and 4-bit weights with memory-mapped per-layer embeddings, and that GPU optimisations process 4,000 input tokens across two skills in under 3 seconds. Constrained decoding enforces structured output, and dynamic context lengths let a single model span CPUs and GPUs. On smaller hardware, LiteRT-LM reaches 133 prefill and 7.6 decode tokens per second on a Raspberry Pi 5 CPU, and 3,700 prefill and 31 decode tokens per second on the NPU of a Qualcomm Dragonwing IQ8, the processor behind the Arduino VENTUNO Q. Gemma 4 is available with CPU and GPU support on Android and iOS, system-wide on Android through the new AICore Developer Preview, on Windows, Linux and macOS via Metal, and in the browser through WebGPU. Google also released a Python package and a litert-lm CLI for Linux, macOS and Raspberry Pi, with tool calling included. Source: https://developers.googleblog.com/bring-state-of-the-art-agentic-skills-to-the-edge-with-gemma-4/ Read the article: https://llmobile.kavents.com/ticker/gemma-4-edge-agent-skills/GoogleGemmaAgentsOpen weightsLiteRThttps://llmobile.kavents.com/ticker/clawmobile/https://llmobile.kavents.com/ticker/clawmobile/Thu, 26 Feb 2026 13:34:00 +0100Seven researchers at MBZUAI, City University of Hong Kong and one working independently published ClawMobile on February 26, 2026, an agent runtime that runs on the Android phone itself with no tethering to a host machine. The model is not on the device. The authors state that ClawMobile runs locally while model inference is performed remotely, and all agents in their tests use GPT-5.2, so the split is an on-device runtime driving a cloud model. Across six real-life phone tasks the authors report ClawMobile completing all six at 100 percent, against 33 to 100 percent for the UI agent they compare against. What separates it from screen-reading GUI agents is that the reasoning loop never touches the screen. The orchestrator issues tool calls to control backends instead, and the authors describe a deterministic-first policy in which the runtime first checks its memory for a structured interface, reaching for ADB system commands and the Termux hardware API before anything visual. A semantic UI agent is invoked only when a step genuinely depends on reading dynamic interface state, and raw taps on interface elements sit below that as a fallback. After every action the runtime re-queries device state to confirm the step landed, which the authors contrast with inferring execution success purely through model reasoning. Architecture diagram: Du et al., CC BY 4.0. The tests ran on a Google Pixel 9 on Android 16, with DroidRun as the UI agent baseline and a stripped ClawMobile without DroidRun as a second baseline, all three on GPT-5.2 and scored by human annotators. In the authors’ table DroidRun reaches 100 percent on switching the system to dark theme and on playing a YouTube video, 85 percent on posting a YouTube comment, 73 percent on a Chrome search and on a cross-app task that searches football results and writes them into Notes, and 33 percent on installing an app from the Play Store, where ClawMobile scores 100 percent throughout. The authors put the cost of that at 57.5 seconds slower per task on average, with ClawMobile taking 21 seconds for the dark theme switch and 235 seconds for the YouTube comment. Their deterministic-only variant hits the 600-second timeout on both YouTube tasks. The authors name the remote model as the main open problem, writing that fully remote inference introduces network latency and privacy concerns and calling on-device inference a promising direction they have not taken. They also flag that serialising full UI trees or screenshots into model context can dominate token use and latency on mobile, and present the results as preliminary, six tasks on one device. The code is open source on GitHub, built on the OpenClaw agent framework with a Telegram bot as the chat channel, and the paper was accepted at EuroMLSys 2026 under a Creative Commons Attribution 4.0 license. Source: https://arxiv.org/abs/2602.22942 Read the article: https://llmobile.kavents.com/ticker/clawmobile/AgentsAndroidPixelResearchOpen sourcehttps://llmobile.kavents.com/ticker/mobile-agent-v3-5/https://llmobile.kavents.com/ticker/mobile-agent-v3-5/Sun, 15 Feb 2026 02:52:00 +0100Alibaba’s Tongyi Lab published Mobile-Agent-v3.5 on February 15, 2026, describing GUI-Owl-1.5, a family of agent models that read a screenshot and emit the next tap, type or scroll. The family spans 2B, 4B, 8B, 32B and 235B-A22B parameters in instruct and reasoning variants, all built on Qwen3-VL. Every benchmark in the paper was run server-side against cloud sandboxes and emulators, and the authors give no on-device deployment, latency or hardware figures anywhere, so the small sizes are the only thing that points at a phone. The authors write that the smaller instruct variants, which skip the reasoning step, are fast enough to be deployed on edge devices and to work alongside a larger cloud model. Across platforms, the authors report their 32B-Instruct model reaching 56.5 on OSWorld-Verified for desktop tasks, 47.6 on OSWorld-MCP and 46.8 on Mobile-World, the last two of which also require calling external tools. On AndroidWorld, which scores an agent driving real Android apps in an emulator, their 8B reasoning model reaches 71.6, which the paper puts just behind UI-TARS-2 at 73.3. For browsers, the paper’s table has the 32B reasoning model at 48.4 on WebArena, 46.6 on VisualWebArena and 82.1 on WebVoyager. The small checkpoints carry a large part of that. The authors report 2B-Instruct at 43.5 on OSWorld-Verified and 67.9 on AndroidWorld, and 4B-Instruct at 48.2 and 69.8, which puts the 4B model within 2 points of their own 32B on Android tasks. On ScreenSpot-Pro, which tests whether a model can point at the right element in a high-resolution screenshot, they report 72.9 for 32B-Instruct and 80.3 when the model first crops a region and looks again. The paper’s prose and its tables disagree on which variant owns several headline numbers, and the figures above follow the tables. Charts: Alibaba Tongyi Lab, the authors' own benchmark figures. Six checkpoints are on Hugging Face under the MIT license, covering 2B, 4B and 8B instruct, 8B reasoning, and 32B in both variants. The 235B-A22B model named in the paper is not among them. The code is MIT as well, and the repository points at a hosted cloud-sandbox demo rather than a local runtime. Source: https://arxiv.org/abs/2602.16855 Read the article: https://llmobile.kavents.com/ticker/mobile-agent-v3-5/AlibabaAgentsQwenOpen weightsAndroidhttps://llmobile.kavents.com/ticker/apple-foundation-models-framework/https://llmobile.kavents.com/ticker/apple-foundation-models-framework/Mon, 09 Jun 2025 21:00:00 +0200Apple opened its on-device foundation model to third-party apps at WWDC25 on June 9, 2025. The Foundation Models framework gives any app direct access to the roughly 3-billion-parameter model, which runs offline and costs developers nothing per call. Developers annotate Swift data structures with the @Generable macro, and the framework applies constrained decoding, which Apple calls guided generation, so the model returns those structures rather than free text. Tool calling lets the model invoke functions the app supplies. Developers who need more control can train rank 32 adapters with a Python toolkit. Apple compressed the model to 2 bits per weight using quantisation-aware training. It splits the transformer into two blocks at a 5:3 depth ratio and shares the key-value cache between them, which Apple says cuts KV cache memory use by 37.5 percent. Images are handled by a 300M-parameter ViTDet-L vision encoder. The model is designed to support 15 languages. The companion server model on Private Cloud Compute uses what Apple calls a Parallel-Track Mixture-of-Experts design, where independent transformer tracks cut how often the tracks have to synchronise with each other. Diagram: Apple. The server model, not the on-device one. Source: https://machinelearning.apple.com/research/apple-foundation-models-2025-updates Read the article: https://llmobile.kavents.com/ticker/apple-foundation-models-framework/AppleiOSDeveloper toolsQuantisationAgentshttps://llmobile.kavents.com/ticker/phonelm/https://llmobile.kavents.com/ticker/phonelm/Thu, 07 Nov 2024 03:19:00 +0100Researchers at Beijing University of Posts and Telecommunications published PhoneLM on November 7, 2024, a small language model family in 0.5B and 1.5B sizes built in an unusual order. The team first searched transformer configurations for the fastest inference on a specific phone, then pre-trained the configuration that won, instead of choosing a shape for capability and optimising speed after the fact. PhoneLM-1.5B decodes at 58 tok/s on the CPU of a Xiaomi 14 with a Qualcomm Snapdragon 8 Gen 3. The search varied the layer count from 15 to 25, the ratio of intermediate to hidden size between 2 and 5 and the attention type, and the paper lists measured throughput for seven candidate 1.5B shapes on that phone. The winner has 19 layers and a hidden size of 2,560, a hidden size to layer ratio of 134.7 against the 54.6 to 88.6 the authors observe in recent small models. It uses ReLU rather than the SiLU activation most current small models adopt, which the authors argue is cheaper on NPUs built for integer arithmetic and leaves more zeros in the feed-forward output for sparse-computation tricks. The paper notes that only the 1.5B model went through the search. Across seven zero-shot tasks including HellaSwag, PIQA, BoolQ and both ARC sets, the benchmark table puts PhoneLM-1.5B at an average of 67.31%, ahead of SmolLM-1.7B at 66.49% and StableLM 2 1.6B at 66.34% and behind DCLM-1.4B at 68.33% and Gemma 2 2B at 71.31%. PhoneLM-0.5B averages 60.43% against 60.17% for SmolLM-360M and 56.46% for OpenELM-450M. The authors measure PhoneLM-1.5B decoding 1.2 times faster than StableLM 2 1.6B and 1.6 times faster than SmolLM 1.7B on the same handset. The two models saw 1.1T and 1.5T training tokens, all of them from open datasets such as DCLM-baseline, StarCoderData and Dolma. The group ran every device measurement through its own inference engine mllm, with the linear and embedding weights quantised to 4-bit and four threads pinned to the phone’s performance cores. On the NPU it goes through Qualcomm’s QNN framework, where the authors report prompt processing for PhoneLM-1.5B at 654 tok/s on the Xiaomi 14 against 602 tok/s for Qwen 2.5 1.5B. The 4-bit mllm build the group publishes is a 1.41 GB file. The paper states no peak memory figure. An Android demo app on top of mllm runs PhoneLM-1.5B locally for chat and for Android intent invocation, so a user who asks to be woken at 8 o’clock has an alarm set through the system’s own alarm intent. That variant, PhoneLM-1.5B-Call, was fine-tuned with LoRA on DroidCall, 10,000 synthetic function-calling examples covering alarms, timers, email drafts and searches, and the paper places it above GPT-4o-mini, which scores 71.0% on the same test set. Weights are on Hugging Face under Apache 2.0 and the data preparation, training, fine-tuning and evaluation code is on GitHub under GPL-2.0, alongside the list of open datasets used at each stage. Source: https://arxiv.org/abs/2411.05046 Read the article: https://llmobile.kavents.com/ticker/phonelm/ResearchOpen weightsAndroidNPUAgentshttps://llmobile.kavents.com/ticker/ministral/https://llmobile.kavents.com/ticker/ministral/Wed, 16 Oct 2024 10:00:00 +0200Mistral AI introduced Ministral 3B and Ministral 8B on October 16, 2024, two models the company built for on-device computing and at-the-edge use and announced together as les Ministraux. Mistral says Ministral 3B already outperforms its own Mistral 7B on most benchmarks, and reports the base model scoring 60.9 on MMLU against 52.4 for Gemma 2 2B and 56.2 for Llama 3.2 3B. Both models handle up to 128k tokens of context, which the announcement says was capped at 32k when run on vLLM. Ministral 8B uses what Mistral calls an interleaved sliding-window attention pattern, where a layer attends only to a limited span of recent tokens instead of the whole sequence, which the company says makes inference faster and lighter on memory. Mistral reports the instruction-tuned Ministral 3B at 8.1 on MTBench, 64.3 on Arena Hard and 77.4 on HumanEval, the last of those above Gemma 2 2B at 42.7 and Llama 3.2 3B at 61.0. Ministral 8B reaches 8.3 on MTBench and 70.9 on Arena Hard in the same table, while Gemma 2 9B stays ahead on WildBench at 43.8 against 41.3. All of these are Mistral’s own numbers, produced with an internal framework the company says it used to re-evaluate every model for the comparison. Table: Mistral AI's own figures, from its internal evaluation framework. Mistral names on-device translation, smart assistants that work without an internet connection, local analytics and autonomous robotics as the applications customers and partners asked for, citing demand for local, privacy-first inference. The company also positions the two models as intermediaries in front of larger ones such as Mistral Large, tuned to parse input, route tasks and call APIs across multiple contexts in multi-step agent workflows. Both models went live through Mistral’s API on the day of the announcement, at $0.04 per million tokens for Ministral 3B and $0.1 per million for Ministral 8B, with input and output priced the same. Mistral published weights for Ministral 8B Instruct only, for research use, and lists that model under both the Mistral Research License and the Mistral Commercial License, while Ministral 3B appears under the commercial license alone with no weights release. Anyone wanting to self-deploy either model has to contact Mistral for a commercial license, and the company says it will help with lossless quantisation for specific use cases. Source: https://mistral.ai/news/ministraux/ Read the article: https://llmobile.kavents.com/ticker/ministral/MistralBenchmarksTranslationAgentshttps://llmobile.kavents.com/ticker/octopus-v3/https://llmobile.kavents.com/ticker/octopus-v3/Wed, 17 Apr 2024 17:07:06 +0200Wei Chen and Zhiyuan Li posted Octopus v3 on April 17, 2024, a technical report on a multimodal agent model that takes an image alongside a text query and answers with a function call. The model has fewer than 1B parameters, handles English and Chinese in one set of weights, and is built to run on edge devices. The paper lists both authors at Stanford University and gives a Nexa AI address for correspondence. The model encodes each supported function as a single functional token in its vocabulary, learned the way a language model learns an unfamiliar word from the contexts it appears in, so the output is one token for the action plus its arguments and needs no parser to read. Images run through a CLIP-based encoder, which the authors say they chose after comparing encoding methods. Training runs in stages, with the language model and the image encoder trained apart, then merged and aligned, then taught the functional tokens using the method from Octopus v2, and finally tuned by reinforcement learning with another large language model acting as the reward model. The evaluation consists of 10 smartphone functions converted into functional tokens and ten worked examples, each printing the model’s output next to the output of GPT-4V feeding GPT-4, which the authors prompted with all the function descriptions and a few examples. The authors write that their model performs comparably to that pair on the selected functions. The report carries no accuracy rate, no success rate and no benchmark table, so the comparison rests on the ten printed generations rather than on a measured score. The report states that the model runs on devices as constrained as a Raspberry Pi and that it makes converting a Raspberry Pi into something like a Rabbit R1 or a Humane AI Pin practical with an on-device model instead of a cloud service. It names no Raspberry Pi board, no phone and no other hardware it was run on, and gives no latency, no memory footprint and no throughput, so the edge deployment is asserted rather than shown. The authors do name one cost themselves, writing that vision input may introduce considerable latency and that they were optimising inference speed. Weights and inference code were available only on application when the report went up, which describes the model as still under testing and for research purposes only, and the authors state that the functional token is licensed. The project page that hosted the demo videos no longer resolves. The authors list audio and video as the modalities they want to support next. Source: https://arxiv.org/abs/2404.11459 Read the article: https://llmobile.kavents.com/ticker/octopus-v3/Nexa AIAgentsResearchhttps://llmobile.kavents.com/ticker/octopus-v2/https://llmobile.kavents.com/ticker/octopus-v2/Tue, 02 Apr 2024 11:01:00 +0200Wei Chen and Zhiyuan Li published Octopus v2 on April 2, 2024, a 2B function-calling model fine-tuned from Google’s Gemma 2B that turns a spoken or typed request into an Android API call. On their own evaluation set built from 20 Android APIs, the authors report 99.524% accuracy and 0.38 seconds per call for Octopus v2, against 98.571% and 1.02 seconds for GPT-4. Both authors give Stanford University as their affiliation on the paper and publish the weights under the name Nexa AI. The distinguishing idea is what the authors call a functional token. Each of the 20 APIs gets its own special token added to the tokenizer and to the model’s output head, so picking a function becomes a single choice among 20 token candidates instead of writing the function name out as text and hoping every piece of it lands correctly. Since the model learned during fine-tuning what each of those tokens stands for, the function descriptions no longer have to sit in the prompt, which the paper puts at over 95% less context. Retrieval-augmented function calling processes roughly 1,000 tokens per call in their comparison, while Octopus v2 sees the query alone, typically under 30 tokens. Shorter context is also where the speed comes from. The authors measure Llama-7B with retrieval-augmented function calling at 68.095% accuracy and 13.46 seconds per call on a single NVIDIA A100, and Octopus v2 at 0.38 seconds under the same settings on the same A100, a 35-fold difference. The phone figure is a separate measurement. With the model quantised and the state for the fixed prompt prefix precomputed, they report a function call finishing in 1.1 to 1.7 seconds for queries of 20 to 30 tokens on what the paper calls a standard Android phone, without naming the device, and they estimate 37 times more calls per iPhone battery charge than a 7B model with retrieval. The authors name limits of their own result. Training with LoRA instead of the full model drops accuracy to 99.048%, and cutting the training set from 1,000 to 100 generated examples per API drops it to 98.095%, which they still put forward as the low end of a usable range. The set of callable functions is fixed at training time, since every API has to exist as a token before the model can emit it, and they price the data generation at 0.0224 USD per 1,000 examples per API. They also note that the GPT-3.5 and GPT-4 latencies were taken through OpenAI’s API on March 18, 2024, and may reflect API traffic or hardware allocation at that moment. Nexa AI released the weights on Hugging Face under a non-commercial CC BY-NC 4.0 license, on top of Google’s Gemma 2B. The model card said at the time that the team was still preparing the code for an open-source release. Source: https://arxiv.org/abs/2404.01744 Read the article: https://llmobile.kavents.com/ticker/octopus-v2/Nexa AIAgentsAndroidGemmaResearchhttps://llmobile.kavents.com/ticker/octopus-function-calling/https://llmobile.kavents.com/ticker/octopus-function-calling/Tue, 02 Apr 2024 03:29:28 +0200Wei Chen, Zhiyuan Li and Mingyuan Ma of Stanford University and Harvard University posted Octopus on April 2, 2024, four open models fine-tuned to pick the right software API for a request and fill in its arguments. On a benchmark the authors built themselves, they report their fine-tuned Gemma 2B answering 93 percent of calls correctly against 96 percent for GPT-4 and 50 percent for GPT-3.5, with the CodeLlama 7B and Gemma 7B versions reaching 97 percent once an output filter is switched on. The title says on-device, but nothing in the paper runs on a device. The authors fine-tuned the models on A100 80GB GPUs and queried GPT-4 and GPT-3.5 through OpenAI’s API, so every number was measured server-side. The device claim is a single sentence stating that models of this size can already be deployed on mobile, pointing at MLC LLM, and the paper names no phone, no chip and no latency, memory or throughput figure. The training set starts from roughly 30,000 of the most used APIs on RapidAPI Hub, which GPT-4 cut to about 20,000 by dropping entries with missing arguments or descriptions that did not match their parameters. The authors rewrote each API as a Python function signature with a docstring, had GPT-4 write five answerable queries per API and an equal share of unanswerable ones, and let GPT-4 verify the result, ending at about 150,000 examples as described in the paper. Vector search over the function descriptions pulled in near-identical functions as distractors, and training worked through them from easy to hard. Base models were CodeLlama 7B, Gemma 2B and 7B and Stable Code 3B, all tuned with LoRA at rank 16 and 8-bit quantisation, using 90 GPU hours for CodeLlama 7B and Gemma 7B, 60 for Stable Code 3B and 30 for Gemma 2B. The output filter the authors call a conditional mask blocks, at each generated token, anything that would break the expected format, so a function name has to come from the known list and an argument has to match its declared type. The authors state that it cannot be applied to GPT-4 or GPT-3.5 because the OpenAI API does not return the underlying token scores, and that GPT-3.5 was not counted wrong for formatting mistakes. The paper states that the training dataset and the fine-tuned models will be open sourced soon, and names no repository or date. The preprint went up under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license. Source: https://arxiv.org/abs/2404.01549 Read the article: https://llmobile.kavents.com/ticker/octopus-function-calling/Nexa AIResearchAgentsGemmaBenchmarks