Vercel Labs ships agent-desktop 1.0.2 as open source under the Apache-2.0 license
Vercel Labs has released agent-desktop as open source: a Rust tool that lets coding agents drive native apps on macOS, Windows, and Linux through the operating system's accessibility tree instead of screenshots. The company itself reports token savings of 78–96 percent — but the figures remain independently unverified.
Version 1.0.2 of agent-desktop launched on October 30, 2026, under the Apache-2.0 license, according to MarkTechPost, which covered the release on October 31, 2026. The tool targets the growing "computer use" category — AI agents that operate a computer the way a user would. Where the model-native approaches from Anthropic and OpenAI rely on screenshots and pixel coordinates, agent-desktop reads each app through the operating system's accessibility tree and gives the agent a structured JSON overview with a reference to every element it can act on.
A tool, not an agent
agent-desktop consists of a CLI interface and a C library written in Rust. It is worth emphasizing what it is not: the tool contains no language model and has no graphical interface. It is a tool that an agent calls — typically a shell-capable coding agent — and each command prints one JSON envelope and exits. The agent itself, and the choice of model, sit outside the tool. That makes agent-desktop model-agnostic: in principle, the same tool can be used with any model that can run shell commands.
How it works
The core, agent-desktop-core, reaches the operating system exclusively through a PlatformAdapter abstraction. In practice, that means the tool speaks to each platform's native accessibility API: AX API on macOS, UI Automation on Windows, and AT-SPI on Linux. The accessibility tree is the same structure screen readers use, and it describes an app's elements — buttons, fields, menus — with roles and states rather than pixels.
When an agent takes a "look" at an app, it therefore receives not an image but a JSON snapshot with a stable reference (ref) to each actionable element. The agent can then perform actions against those references. By default, ref actions go through the accessibility API without moving the mouse pointer or taking focus — this is what makes it possible for the agent to work while the user keeps using the machine. A global --headed flag allows physical input when needed. Each ref action auto-waits up to 5,000 milliseconds for the target to be visible, enabled, and stable.
The failure model: fail closed
agent-desktop chooses a strict failure model: ref resolution is strict and fails closed. When the UI changes between snapshot and action, the tool returns a STALE_REF or AMBIGUOUS_TARGET error, and the agent must take a new snapshot. Every error carries a disposition, and the agent should only retry when the disposition says safe. The design thus prioritizes explicit errors over assumed successes.
The math: tokens versus pixels
The core argument for the approach is cost. According to Anthropic's own documentation, cited by MarkTechPost, each screenshot in Claude computer use costs roughly 1,000–1,800 input tokens, and the toolset definition itself adds around 4,500 tokens per request. Over a long task with many screenshots, that adds up quickly.
The publisher's own figures point in the same direction, but must be read with caveats. The tool's developer reports token savings of 78–96 percent compared with full dumps of the accessibility tree in dense Electron apps such as Slack, VS Code, and Notion. In Apple Maps, a single look reportedly fell from 3,811 to 273 tokens. In a simulated 52-step task, consumption reportedly dropped from 198,000 to 31,000 tokens. All of these figures are self-reported by the builder — there is as yet no independent benchmark confirming them.
Others are already using the same underlying technique. Microsoft's UFO² also uses UI Automation on Windows, but ships as a full-fledged Windows agent — a far more extensive system — while agent-desktop deliberately stays a single, model-agnostic tool.
Open questions
All coverage of the release stems from a single secondary source, MarkTechPost; neither the repository, changelog, nor any announcement from Vercel Labs itself is available in the current evidence base. The performance figures are, as the source itself notes, self-reported. A comparison table against Claude computer use, OpenAI computer use, and UFO² is not fully documented in the source and therefore cannot be verified here.
Another unanswered question is the security profile: what it means to let an agent control native desktop apps headlessly — without pointer movement or focus — is not addressed in the source, and requires independent coverage before conclusions can be drawn. The accessibility tree gives an agent tight integration with the user's actual machine, and that is a tradeoff that deserves its own analysis.
What to watch
The interesting future development is twofold. First: independent benchmarks that can confirm or refute the token savings, and primary sources — a repository and documentation — that can verify the technical details. Second: how the relationship between structured snapshots and screenshots evolves — structured data where it exists, screenshots where it does not. For now, the evidence supports a tradeoff between the approaches, not a verdict on either.