CAPED redacts phone screenshots before a cloud GUI agent sees them
Researchers at the Chinese University of Hong Kong, Southeast University and Shandong University published CAPED on June 10, 2026, a filter that blacks out parts of a phone’s screenshots before they reach a cloud-hosted GUI agent. Across a 28-task privacy suite the authors built themselves, weighted seeded leakage falls from 0.766 with raw screenshots to 0.268 with the filter in front, counted only on runs where the agent still finished its task. A multimodal model plays the attacker and scores what it can still infer from the whole screenshot sequence, not from single frames.
A screenshot-based agent operates a phone the way a person does, by looking at the screen, so every image it uploads carries whatever happens to sit next to the thing the user asked for. The authors call this incidental visual privacy exposure and name contacts, message previews, non-target photos, filenames, recommendation feeds and health widgets as content the remote model receives although the user never mentioned it. In one of their test shops, a request to buy sunglasses puts prenatal vitamins, a baby bottle set and a maternity dress in front of the model, because the shop’s recommendation rows share the page with the product. Stripping names and numbers out of the text does not close that gap, and the authors measure text-only anonymisation at 0.752 against 0.766 for no protection at all.
CAPED decides per interface element rather than per screen, according to the paper. It first sorts the current screen into one of three postures, so galleries, maps, files and financial views show their text but hide their images, while messaging, contacts, feeds and health dashboards hide both by default. The user’s instruction is turned into a short list of what the agent is allowed to need, and a hidden element is released only when it matches that list, checked either by text matching or by asking a vision model whether a cropped thumbnail really shows the requested subject. Hidden areas keep their place in the layout under a solid overlay with an eye icon, and names in exposed text are replaced by placeholders the remote agent can still follow.
The defence is meant to run on the phone, and the authors argue the instruction has to be interpreted locally too, since it can name people, relationships, locations and intentions before any screen policy applies. The prototype they measured does not run on a phone. They state that CAPED’s components ran on the evaluation host with an evaluation-side service for the heavier models, and that the timings should be read as live-loop prototype latency rather than physical-phone latency, energy or thermal behaviour. Over 1,394 measured steps the privacy work added 2.325 seconds per agent step, 22.6 percent of a 10.835-second step, most of it the detector at 1.811 seconds, plus 5.716 seconds once per task to extract the requirements.
The protection also costs task success. On AndroidWorld’s 116 ordinary tasks, GUI-Owl-7B completed 77 of them (66.4%) unprotected and 64 (55.2%) behind the full filter, which the authors call a prototype-level utility cost. They name further limits themselves, writing that screens their rule base does not cover fall back to the most permissive posture, which they describe as not privacy-conservative, and that open-ended requests such as finding the latest receipt widen exposure and account for much of the leakage that remains. On the connected Gmail, Drive and Photos tasks the filter moves leakage from 0.402 to 0.347 only, the interactive unmasking path was never tested with real users, and the authors write that they plan to release the CAPED artifact in a future public version.