Desktop Pilot MCP

Native macOS automation for Claude. 30-100x faster than screenshots.

Desktop Pilot is an MCP server that gives Claude direct access to any macOS application through the Accessibility API, AppleScript, and CGEvent -- no screenshots, no pixel coordinates, no vision model overhead. It reads the actual UI tree and acts on semantic element references, the same way Playwright works for browsers.

One snapshot of Telegram takes 20ms and returns structured data. The same operation with screenshot-based computer-use takes ~3 seconds and returns pixels.

pilot_snapshot { "app": "Telegram" }

[e1] Window "Saved Messages"
  [e2] MenuButton "Main menu"
  [e3] Button "All chats (111 unread chats)"
  [e7] Button "Code (4 unread chats)"
  [e18] TextField "Write a message..."
  [e20] Button "Record Voice Message"

pilot_click { "ref": "e18" }       // focus the text field
pilot_type  { "ref": "e18", "text": "Hello from Claude" }
pilot_click { "ref": "e20" }       // send

No coordinates. No screenshots. No guessing. Just refs.

Quick Start (2 minutes)

npx desktop-pilot-mcp

Step 1. Add to your Claude config and restart Claude:

For Claude Code, add to ~/.claude.json under your project's mcpServers:

{
  "desktop-pilot": {
    "command": "npx",
    "args": ["-y", "desktop-pilot-mcp"]
  }
}

For Claude Desktop, add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "desktop-pilot": {
      "command": "npx",
      "args": ["-y", "desktop-pilot-mcp"]
    }
  }
}

Step 2. Grant Accessibility permission when macOS prompts you (one-time).

If the prompt doesn't appear: System Settings > Privacy & Security > Accessibility -- add your terminal app or Claude Desktop.

Step 3. Ask Claude to interact with any app:

"Take a snapshot of Telegram and show me what's on screen"

That's it. No API keys, no accounts, no configuration files.

git clone https://github.com/VersoXBT/desktop-pilot-mcp.git
cd desktop-pilot-mcp
swift build -c release

{
  "desktop-pilot": {
    "command": "/absolute/path/to/desktop-pilot-mcp/.build/release/desktop-pilot-mcp",
    "args": []
  }
}

Benchmarks

Real measurements from testing against Telegram, Finder, and other macOS apps:

... View full README on GitHub