Vercel agent-desktop: computer use through the accessibility tree, not screenshots, and what it still needs from AUX

Figure 1 · Two ways of seeing a screen

One guesses from a picture; the other reads the app’s own structure

Two tracks

Who this is for, and how to read it

Two audiences read auxfirst.com, so this piece runs on two tracks. Business track sections explain what the tool changes for teams that run on desktop software, what it costs, where the risk sits, and which decisions remain yours. Technical track sections go into the architecture, the observe-act-verify loop, error semantics, the MCP server, the C library, and how to run it anywhere, including far from Vercel’s cloud.

Both tracks meet in section 08, where we hold agent-desktop up against what auxfirst preaches: the Action Heat Ladder, agent-first design, the four kinds of trust in Agentic User Experience, and the question we keep asking on behalf of brands: can I prove what the agent did?

Business track

What agent-desktop is

Every modern operating system keeps a hidden, structured description of what is on screen. It exists so that a screen reader can tell a blind user “Button: Send” instead of “a blue rectangle at pixel 812, 44”. That description is the accessibility tree. agent-desktop reads it and hands it to an AI agent as a list: this is a text field called Subject, this is a button called Send, this is a checkbox called Remember me, and it is currently unchecked. Each item carries a short handle, a “ref”, the agent can act on.^[1]

Three consequences follow, and they are the whole business case.

1. No guessing from pictures. The agent does not look at a screenshot and estimate where “Send” is. It acts on the element the app itself calls “Send”. Theme changes, window moves, high-DPI screens and dark mode stop mattering.
2. No stolen mouse. By default, actions are “headless”: they go through the accessibility API without moving your cursor or taking keyboard focus.^[1] A person can keep working in another window while the agent fills in a form behind it.
3. Far fewer tokens. Reading a tree of names is much cheaper than sending a model a full-resolution screenshot every step. The builder’s own figures, as reported by CellCog from his launch post: a single look at Apple Maps dropped from 3,811 tokens for the full window to 273 for a skeleton-and-drill read, and a simulated 52-step task from about 198k tokens to about 31k.^[7] Treat these as vendor figures until an independent benchmark exists; there is none yet.

What it is not: it is not an agent, has no model inside, has no GUI, and does not automate web pages; that is the job of its sibling, agent-browser.^[1][5] It is a tool an agent calls, one command at a time. The thinking, what to do, when to stop, when to ask a human, lives in whatever drives it: Claude Code, Codex, Cursor, a custom script, or your own agent runtime.

The facts a buyer needs, in one block

Sources for this table: the repository README, CHANGELOG, Cargo manifest and release list; the GitHub API for star and fork counts; DEV Community, 2 November 2026.^[1][2][8]

Business track

Why it matters now

Three things converged in late October 2026.

First, “computer use” stopped being a demo. Anthropic, OpenAI and Microsoft all ship agents that can operate a screen. Almost all of them see the screen as pixels: a screenshot goes to a vision model, the model returns coordinates, the harness clicks there. That works, but it is slow, expensive and brittle. Anthropic’s own documentation puts one screenshot at roughly 1,000 to 1,800 input tokens, before the model has reasoned about it.^[6] Vercel’s bet is that for the large majority of business software, the accessibility tree already contains everything the agent needs, at a fraction of the cost.

Second, the long tail of enterprise work still runs on desktop software. ERP clients, accounting packages, claims systems, lab software, hospital front-ends, trading terminals, design tools, legacy Win32 and WPF line-of-business apps: a vast amount of knowledge work happens inside windows no browser agent can reach. This is where robotic process automation (RPA) vendors have earned their keep for fifteen years, by scripting clicks against those same accessibility trees. agent-desktop puts that capability in an open, model-agnostic, scriptable tool that any LLM harness can drive.

Third, Vercel is a credible open-source maintainer with distribution. The repository roughly doubled its stars in the three days after launch. The project ships a documentation site, probe corpora with findings ledgers per platform, a C ABI with versioned headers, and an npm package that bundles native binaries.^[1][3] This is not a weekend hack; it is nine months of releases that got a very large megaphone.

Figure 2 · Timeline

From one developer’s repository to a three-platform 1.x release in nine months

Technical track

How it works

The seven-layer chain

The documentation describes one chain from caller to app, and the code matches it:^[3]

1. Your agent: a model loop or script. It runs a command and reads one JSON envelope.
2. The agent-desktop CLI, or the C library when called in-process.
3. agent-desktop-core: commands, the RefStore, snapshot building, and every shared type.
4. PlatformAdapter: a Rust trait, the only door from core to an operating system.
5. One platform crate, macOS, Windows or Linux, chosen at compile time.
6. The OS accessibility API: AX, UIA or AT-SPI.
7. The app, which exposes its UI through that API.

Core never imports a platform crate; platform crates never import each other. A binary drives exactly the OS it was built for. This is the kind of boundary discipline that makes a tool portable and auditable: you can read the core crate and know that every OS interaction passes through one trait.

Figure 3 · The chain

Seven layers, one door to the operating system

The loop in four commands

agent-desktop snapshot --app "Notes" --skeleton -i --compact   # 1. map: shallow skeleton, interactive nodes only
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # 2. drill: expand one region into the same snapshot
agent-desktop click @s8f3k2p9:e12                              # 3. act on a qualified ref
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # 4. verify by reading, not by trusting ok:true

Every command prints one JSON envelope: ok, command, and either data or error. A trimmed snapshot, from the README:^[1]

{
  "version": "2.6", "ok": true, "command": "snapshot",
  "data": {
    "app": "Finder", "snapshot_id": "s8f3k2p9", "ref_count": 59, "complete": true,
    "tree": { "role": "window", "name": "Downloads", "children": [
      { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] }
    ] }
  }
}

The design choices that make this trustworthy, in order of importance:

Refs are qualified and short-lived. A ref is @<snapshot_id>:e<N>. A bare @e8 is rejected with INVALID_ARGS. Refs are valid for one snapshot; after any UI change the agent re-reads. On macOS a ref carries the element’s own accessibility address, so identical twins each keep their ref. On Windows and Linux a ref resolves only if it was unique in the window when captured (by role, native id, stable name and description) and exactly one live element still matches. Zero matches is STALE_REF; two or more is AMBIGUOUS_TARGET. The tool refuses to guess.^[2][3]

Skeleton and drill-down keep token cost down. --skeleton reads only a few levels deep; cut containers report a child count and get a drill ref. snapshot --root <ref> then reads one region into the same snapshot id. A “web wrapper” rule skips the anonymous container chains Electron and Chromium apps are made of, so a dense web app fits a default depth budget instead of understating its content by an order of magnitude.^[3]

Evidence is tri-state. Every property an adapter reads is Known, Absent or Unknown, and the three are never collapsed. Absent means the provider was asked and does not have it. Unknown means the read failed. Only a positive claim grants a role, state or action.^[3] This sounds academic until you realize it is the difference between “the Send button is disabled” and “I could not tell whether the Send button is disabled”, a distinction most automation stacks throw away.

Actionability, delivery and retry: the honest-failure model

Before any action dispatches, core runs an actionability battery against the live element: visible, stable, enabled, supports the action, allowed by policy, editable, and actually receiving events (a five-point hit test, so a partially covered control still passes). Transient checks are polled for up to 5,000 ms by default; terminal checks fail fast.^[3]

Then comes the part every agent builder should steal. Every action result and every error carries a disposition:^[4]

The rule the bundled skill gives agents is blunt: Retry or substitute an action only when disposition.retry is safe. Only not_delivered is safe.^[2] A repeated click can submit a form twice; a repeated toggle undoes itself. Where the tool cannot prove whether a click landed, it says so and tells the agent to go and look. There are 16 error codes, each with a documented “what to do”, and a recovery object (refresh_snapshot_then_retry_original and friends) appears only when retrying is safe.^[4]

On Windows this honesty goes further. The SendInput API reports success whether or not User Interface Privilege Isolation blocked the input, so agent-desktop never treats that return value as delivery evidence. It compares process integrity tokens first and refuses with PERM_DENIED when the target runs elevated.^[3][9]

Headless by default, headed by permission

Ref actions are headless: semantic accessibility calls with no focus steal, no cursor movement, no synthetic keystrokes, no clipboard. If no semantic path exists, the action fails closed with POLICY_DENIED. The global --headed flag is a permission to use physical input (real clicks, real keys), and the skill tells agents: Never switch to headed mode to recover from a headless failure.^[2] Which commands need --headed differs by OS. On Windows, for example, type has no semantic path, so set-value is the headless way to write text; double-click, hover, drag and right-click are physical everywhere.^[3][9]

A machine-wide interaction lease serializes physical input, so two agents on one machine cannot interleave keystrokes. Headless semantic actions do not need the pointer at all, which is what lets several agents work in parallel on one desktop.^[3]

Two safety rails that come for free

• Secure fields. Once the OS marks a field as a password field (IsPassword on Windows, the AT-SPI password role, macOS secure text), agent-desktop withholds its value on the read side and never echoes an attempted or observed value on the action side: not in messages, not in details, not in post-state. It can type into a password field; it cannot leak one.^[3]
• Protected processes. It refuses to close session-critical processes (Finder, explorer.exe and the like) on every surface, with exact-name matching so look-alikes stay closable.^[3]

Sessions, traces and the cursor overlay

• A session (session start) is an on-disk container holding snapshot refmaps, a session.json manifest and a trace/ directory. Lookups are confined to the session namespace.^[3]
• Traces are append-only JSONL files, one per OS process, each opening with a trace.meta header; trace show merges them into a deterministic timeline and trace export writes HTML. The schema is additive-only. Opt-in replay artifacts store pre- and post-action PNGs, unredacted, so the documentation tells you to treat them like screenshots.^[3]
• The agent cursor overlay draws a labeled, colored cursor that moves like a human hand to each verified action target, plays a ripple on click, and shows a caret after typing. It never moves the real pointer and never changes what the action does. With --multi-agent each sub-agent gets its own labeled cursor.^[3] This is presentation only, but it is the first time we have seen a computer-use tool ship a disclosure affordance for the human watching.

Figure 4 · The cursor overlay

Every agent is visibly on screen; none of them touch your mouse

The agent learns the tool from the binary

agent-desktop skills get desktop prints the core guide (the loop, refs, verification and retry rules); skills get platform prints the OS-specific one. The same skill is published on skills.sh, and the README suggests two lines for your AGENTS.md:^[1]

Use `agent-desktop` to see and operate desktop apps. Before the first use, run
`agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them.

The skills are versioned with the binary, so the agent never reads instructions for a release it does not have. This is the SKILL.md practice done properly: a procedure that loads when the work calls for it, kept in step with the tool it describes.

MCP: every command as a tool, new on 3 November 2026

agent-desktop mcp serves every command as an MCP tool over stdio. Tools are named desktop_ plus the command (desktop_snapshot, desktop_click, desktop_session_start); arguments mirror the CLI flags. The tool list is generated from the CLI’s own definitions, so it always matches the installed binary. Client configuration is one line:^[1][5]

{ "mcpServers": { "agent-desktop": { "command": "agent-desktop", "args": ["mcp"] } } }

Design notes worth knowing before you wire it into a harness:^[5]

• Global flags (--session, --headed, --trace) are set once on the server command line; a tool cannot set them. That is a deliberate control point: the model cannot escalate itself to headed mode.
• Results come back as structuredContent plus text; isError mirrors ok: false, so clients branch on error.code and error.disposition exactly as with the CLI.
• The server runs at most 8 tool calls at once, refuses messages over 4 MiB, and on cancellation sends SIGTERM first so a held key or button is released.
• The security paragraph in the MCP guide is the most important sentence in the whole project: Treat the server like a shell for the signed-in user. Keep your client’s confirmation on for actions, and remember that text read from the screen is untrusted input to the model. More on that below.

C library, for everything that is not a shell

Each release ships libagent_desktop_ffi (dylib, so, dll) with a versioned C ABI, major 7 as of 1.1.0. Python via ctypes, Swift, Go, Ruby and Node hosts can call the same core in-process, with the same strict ref resolution, actionability and policy semantics as the CLI; the project calls this “FFI ref-action parity”.^[1][3]

Electron and Chromium apps: hand the web part to agent-browser

For an app you can launch fresh, launch --cdp starts it with a loopback-only Chrome DevTools port and returns the endpoint. agent-browser, or any CDP client, drives the web contents; agent-desktop keeps the native menu bar, file dialogs, notifications and window management. The two tools are designed as a pair, and the guide is candid that the web contents of such apps can be dense and slow to walk.^[10]

Technical track with a business summary

Running it outside Vercel’s cloud

A common misreading, given the maker: agent-desktop is not a Vercel-hosted service and does not need a Vercel account. It is a native binary that must run on the machine whose desktop it drives, inside an interactive graphical session.^[1][9] That constraint decides where it can and cannot live.

Three deployment patterns we would actually propose

1. Supervised desktop co-worker. Claude Code or Codex on a knowledge worker’s machine, agent-desktop over MCP, the harness confirmation prompt left on, a named session with tracing. Use: data entry into desktop-only ERP or CRM clients, reconciliations across two apps, report assembly from legacy tools.
2. Fleet of unattended runners. A pool of Windows or Linux VMs, each with one interactive session and one agent, multi-agent cursors for the human supervisor’s dashboard, traces shipped to central storage. Use: nightly back-office batches that previously ran on RPA licenses.
3. Desktop app QA in CI. Your own app, driven through the accessibility tree on a macOS or Windows runner, with wait predicates instead of sleeps and traces as the test log. A pleasant side effect: if agent-desktop cannot see a control, neither can a screen reader, so your accessibility backlog and your agent-readiness backlog become the same list.

Business track Technical track

How it compares

How each tool sees the screen is the axis that matters.

Figures from the agent-desktop documentation, Anthropic’s computer-use documentation, the Microsoft UFO repository and the Fazm roundup.^[6][7][13] The honest reading: the accessibility-tree approach wins on cost, determinism and privacy wherever the app publishes a good tree, and loses wherever it does not. The two approaches are complements, not rivals. A mature harness will use the tree first and fall back to vision for the canvas, the game, the remote desktop, and the badly built app.

Where the real differentiation is

Most commentary has focused on the token savings. We think the deeper differentiator is the project’s evidence culture, because it is the thing enterprises will care about once the novelty fades:^[3]

• Every platform adapter was built against a committed probe corpus: raw OS scripts whose redacted captures sit beside the code, cited by row id (A15-7 on Windows, L32-39 on Linux) in a findings ledger that records whether each measurement confirms or contradicts what the project assumed.
• A mandatory measurement gate fails a probe run that declared a measurement and did not produce it, and fails one that declared nothing, because “asserted nothing” is indistinguishable from “all clean”.
• Dogfood runs drive the shipped binary against real software nobody on the project wrote, and report what was not exercised rather than presenting uniform verification.

That is a burden-of-proof discipline most AI tooling simply does not have. It is also, as we will argue, applied only to the tool’s own correctness, not yet to the work the agent does with it.

Business track

Who should use it, and for what

Concrete use cases we would scope first

• Finance back office: posting approved invoices into a desktop accounting client; the agent reads the approved list from one app and types into another, headless, while the controller keeps working.
• Healthcare and life sciences: moving results between a lab information system and a reporting tool where only desktop clients exist, with every write behind a named approver, because the blast radius is a patient record.
• Advertising operations: trafficking approved creative into desktop ad-serving or production tools, with the agent’s cursor overlay visible on a supervisor’s screen and the trace retained per campaign.
• Internal IT: provisioning steps in admin consoles that have no API, run from an elevated session that matches the target’s integrity level.
• Desktop QA: regression suites for Electron, Qt, WPF and native apps that no longer depend on brittle image matching.

Business track Technical track

Where agent-desktop does what we preach, and where it stops

auxfirst exists to design the relationship between people and agents: what an agent may do, when it must ask, what it remembers, how people take control back, and how a business proves what happened. We score tools against that, not against a feature list. Here is agent-desktop on that scale.

Where it does a great job

Figure 5 · Scorecard

Where agent-desktop already behaves the way we ask agents to behave

Fails closed

Refuses stale and ambiguous targets instead of guessing.

Proves delivery

Says whether input reached the app and whether repeating is safe.

Headless by default

Never takes the mouse or the keyboard unless a person allows it.

Protects passwords and the desktop

Password fields never echoed; session-critical processes never closed.

Keeps a trace

Append-only record of every action, with a merged timeline.

Shows the human a cursor

A labeled on-screen cursor per agent, without touching the real pointer.

Teaches the agent its rules

Skills ship inside the binary, versioned with it.

Measures itself

Probe corpora, findings ledgers and gates that fail on silence.

Where it still needs what we preach

All of these are by design. Vercel built a tool, not an agent, and says so. That is the right call for the tool. It is also exactly the gap an organization has to fill before this runs on real work.

Figure 6 · The stack

The tool stops here; accountability starts here

agent-desktop is the best-engineered open primitive for desktop computer use we have reviewed, and the first one whose failure semantics are good enough to build real policy on. Put a mandate, an owner, a heat-scored approval path and a human-readable receipt above it, and you have a desktop agent a business can own. Skip that layer and you have a very efficient way to click “Send” twice.

Business track

A 30-day path to a defensible pilot

1. Pick one desktop-only workflow with cool actions. Reads, drafts, internal-only writes with an undo. Score it on the heat ladder before anyone installs anything.
2. Name the owner. One person with decision rights over the agent’s behavior. Not the vendor. Not “AI”.
3. Install on one endpoint, headless only, with a named session and tracing on. --headed stays off until the mandate explicitly allows physical input for a listed action.
4. Wire it over MCP to a harness that supports per-tool confirmation, and keep confirmation on for every mutating tool. Then graduate: move listed cool actions to auto-run, keep medium ones at propose-and-approve, keep hot ones human-executed.
5. Write the owner’s manual: mandate, allowlisted apps, stop conditions, what the agent must never touch, what to do when a screen contains instructions.
6. Measure supervision cost weekly. Actions taken, actions reviewed, time to review, exceptions. If review time grows faster than value, you have outrun your humans; narrow the mandate.
7. Audit the target apps. If the accessibility tree is poor, fix the app, or choose vision fallback deliberately, before scaling.

Questions people ask

What is Vercel agent-desktop?

An open-source (Apache-2.0) Rust CLI, C library and MCP server from Vercel Labs that lets AI agents read and operate native desktop apps on macOS, Windows and Linux through the OS accessibility tree. It returns the UI as JSON with a ref per actionable element; it contains no model and runs no loop of its own.

Does agent-desktop use screenshots?

Not for targeting. Actions address elements in the accessibility tree. A separate screenshot command exists for humans, debugging and optional replay artifacts, and an app that publishes no accessibility tree is invisible to the tool.

Is agent-desktop a Vercel cloud service?

No. It is a native binary installed on the machine whose desktop it drives, via npm, Homebrew or a release archive. It needs an interactive graphical session, which Vercel Sandbox, serverless functions and plain containers do not provide out of the box. Its sibling agent-browser does have first-class Vercel Sandbox support.

Which AI models work with agent-desktop?

Any. It is model-agnostic: drive it from Claude Code, Codex, Cursor, a local model, or your own agent runtime via the CLI, the MCP server or the C library. The optional JEV helper skill is the only component that calls a remote model endpoint, and it is not part of the installed package.

How does agent-desktop compare to Claude computer use or OpenAI's computer-using agent?

Those see the screen as pixels through a vision model and click coordinates. agent-desktop reads the accessibility tree and acts on named elements, so it is cheaper per step (the builder reports 273 tokens per look instead of 3,811 in one Apple Maps example), deterministic, and does not need to move the mouse. Vision agents still win on apps with no accessibility tree, games, canvases and remote desktops.

What happens when the UI changes mid-task?

Refs are tied to the snapshot that produced them. A changed or removed element returns STALE_REF; a duplicate returns AMBIGUOUS_TARGET. The agent takes a fresh snapshot and acts on new refs. The tool never guesses.

Can two agents share one computer?

Yes. Headless actions do not need the pointer, so several agents can act in parallel. A machine-wide interaction lease serializes physical (headed) input, and the multi-agent cursor mode gives each sub-agent its own labeled on-screen cursor.

Is it safe to point agent-desktop at a company laptop?

Technically, with care: it never echoes password fields, refuses to kill session-critical processes, and runs headless by default. Organizationally, only with a mandate: the tool has no concept of which actions are consequential, text read from the screen is untrusted input to the model, and the trace is an engineering log rather than a business receipt. That governance layer is what auxfirst designs.

What does agent-desktop cost?

Nothing. It is free under Apache-2.0. You pay for the model and the machines you run it on.

What is the difference between agent-desktop and agent-browser?

agent-browser automates web pages and runs in Vercel Sandbox; agent-desktop automates native desktop apps. For Electron and Chromium apps, agent-desktop can launch the app with a DevTools port and hand the web contents to agent-browser while keeping the native UI itself.

Glossary for the business reader

Accessibility tree

The operating system’s structured description of what is on screen (roles, names, states, actions), built for screen readers and now read by agents.

Ref

A short handle, tied to one snapshot, for one element an agent may act on.

Snapshot, skeleton, drill-down

A read of the tree; a shallow read; a deeper read of one region.

Headless versus headed

Acting through the accessibility API without touching the mouse, versus permitted physical input.

Disposition

The tool’s statement of whether an input reached the app and whether repeating it is safe.

Trace and session

The append-only record of what the tool did, and the container that holds it.

MCP

Model Context Protocol, the standard way to expose tools to AI assistants; agent-desktop speaks it natively.

Action heat

auxfirst’s score for how consequential an action is, which sets how much human control it needs.

---