agent-desktop: Let AI agents read and click desktop apps
Added
Made by Vercel Labs
View profile on GitHub (opens in a new tab)For developers building computer-use agents: agent-desktop lets an AI agent read and operate desktop apps on Mac, Windows or Linux from a command line.
What it does
agent-desktop is a command-line tool and C library that lets AI agents use desktop apps. A snapshot command reads an app's accessibility tree (the list of buttons and fields the operating system exposes) and returns JSON, giving each element a reference. The agent acts on a reference, then snapshots again to check the result.
How to use it
Install the npm package globally, then run its version command to confirm it works. On macOS, grant Accessibility permission first. Take a snapshot of an app such as Finder, click a reference from it, and snapshot again to verify. Direct downloads are on the releases page.
Common use cases
- Run a snapshot of the Finder app, and you get JSON listing every clickable element, each with a reference the agent can act on.
- Click a reference from a snapshot and then snapshot again, and you see whether the window changed the way you expected.
- Run the list-apps command, and you see the names of the open apps that you can then snapshot or control.
Can I use this?
Requirements It runs on your own computer, on macOS, Windows or Linux, on x64 or ARM64 processors. Install the npm package, or download a release archive and keep its helper file beside the program.
Caveat Calls are headless by default and do not move the pointer; real mouse input needs the headed flag. When the screen changes, old references fail with a stale-reference error, so take a new snapshot.
Worth knowing Snapshots, clipboard reads and screenshots can contain private desktop content, so keep them out of public logs and uploads. Licensed under Apache-2.0.
Why it's here
- 13,858 people starred it on GitHub this week — 19,335 in total.
Numbers as of 8 November 2026, from BuildTube’s latest daily refresh.
Behind it
View setup guideBuildTube has not run or verified this project. Everything above is written from what the creator published.
Made this?
Quick questions
- What is agent-desktop?
- agent-desktop is a command-line tool and C library that lets AI agents use desktop apps.
- What can I use agent-desktop for?
- Run a snapshot of the Finder app, and you get JSON listing every clickable element, each with a reference the agent can act on. Click a reference from a snapshot and then snapshot again, and you see whether the window changed the way you expected. Run the list-apps command, and you see the names of the open apps that you can then snapshot or control.
- Can I run agent-desktop on my computer?
- It runs on your own computer, on macOS, Windows or Linux, on x64 or ARM64 processors.
- What do I need to use agent-desktop?
- An AI agent or script that can call a command line, plus npm to install. macOS needs Accessibility permission, and Linux needs your graphical session with at-spi2-core.
- What license does agent-desktop use?
- Licensed under Apache-2.0.
More from this week
- OpenBot Run AI coworkers with their own browser and files
Give an AI coworker a job, connect the tools it needs and watch it work.
- Stroq Check AI coding agent sessions for leaked keys
Stroq scans your recent Claude Code and Codex sessions for keys that left your machine.
- DeepTutor Study your own documents with a personal AI tutor
Read documents with page-by-page citations, practice daily and get video lessons from an AI tutor.
Explore by topic
Post this
Download share imageFor makers
Made this? Show it’s on BuildTube — add the badge to your README or embed the card on your site.
README badge (Markdown)
[](https://buildtube.app/p/gh-vercel-labs-agent-desktop/)Embed card (HTML)
<iframe src="https://buildtube.app/embed/gh-vercel-labs-agent-desktop/" width="560" height="240" style="border:0;max-width:100%" loading="lazy" title="agent-desktop on BuildTube"></iframe>