Ghost in the Droid
About
Give any LLM agent a real Android or iPhone as its body — 62 MCP tools for tap, swipe, type, screenshot, app launch, camera, TTS, on-device inference, and batched automation.
Details
- Author
- ghost-in-the-droid
- Categories
- Developer Tools, Other, AI
Jump to
Setup
Install Ghost in the Droid in your MCP client (Claude Desktop, Cursor, Windsurf, and others).
Repository: https://github.com/ghost-in-the-droid/android-agent
Follow the installation instructions in the repository README, then restart your MCP client.
Summon a ghost into your phone.
It sees the screen. It taps the buttons. It never sleeps.
▶Watch full HD with sound on YouTube·Download mp4
Nine agents. Nine real devices. One ghost. (Click to watch.)
Every AI agent can think. Almost none can touch a phone.
Ghost is the body.It gives any LLM agent a real Android phone or iPhone: it reads the screen, taps, swipes, and types through a clean tool surface, and it scales from one device on your desk to a whole phone farm. Point Claude Code, Codex, Antigravity, Cursor, your LangChain app, or a model runninginside the phone itselfat it, and your agent grows hands.
Bring your own brain. Keep the same body. MIT, forever.
Ghost 1.3 is the only Android + iOS agent framework whereplatform,brain, anddriverare all swappable. Pick one from each column. They all compose.
The Docker + KVM emulator pool is Android-only; iOS uses Apple's own simulator and Xcode signing, which are macOS-native.
The body stays the same 62 MCP tools no matter what you plug in. Every new model release is a free upgrade to your phone agent.
Nine demos, one per feature, each recorded on real devices. Click any clip for the full mp4.
Open-source Python framework for controlling Android and iOS devices from one agent harness. Android runs through ADB. iOS runs through Appium XCUITest and WebDriverAgent, with real iPhones as the target path and simulators for development and CI.
Defineskillsfor any app, run them from the dashboard or API, scale across a phone farm.
- Android control through ADB: tap, swipe, type, clipboard, shell, intents, and stealth variants
- iOS control through WebDriverAgent: screenshot, accessibility tree, tap, swipe, type, app launch, clipboard, browser actions
- Live phone screen streaming: Android MJPEG/WebRTC, iOS WDA MJPEG with screenshot fallback
- Interactive touch-to-tap on the streamed screen
- Multi-device phone farm with per-device job queues
- YAML-based UI element definitions per app
- Platform-specific selectors withelements.yamlfor Android andelements_ios.yamlfor iOS
- Python action classes with precondition checks
- Multi-step workflows that chain actions together
- Built-in skills for TikTok and Play Store
- iOS browser/news demo skill and smoke-level TikTok iOS workflows
- Skill Hub: browse, search, and install skills from the community registry
- Install from CLI:android-agent skill install tiktok
- BFS-based auto app explorer: discovers every screen and transition
- LLM-assisted Skill Creator: chat with AI while viewing the live device stream
- The AI identifies UI elements and generates action/workflow code
- Multi-device phone farm with per-device job queues
- Bot runner: queue, schedule, and monitor automation jobs
- Per-device integration tests with Android screen recording or iOS WDA MJPEG recording
Skills cost $0. Thinking costs tokens.An agent should not pay an LLM to tap a button it has tapped a thousand times. Known workflows compile toskills: deterministic YAML + Python recipes that replay in seconds with zero LLM calls. Use AI for the unknown task, replay for the known one.
The phone is the easiest sandbox there is.Give the ghost an old Android with its own SIM and its own accounts, physically separate from your life. You already know how to set up a phone.
Zero-app by default.A fresh install touches nothing on the Android device: screen reads go throughuiautomator, actions throughadb shell input, no root, no accessibility service. Want it faster? The optional Portal companion app gives roughly 30x quicker UI reads. Want it private? On-device mode runs the whole loop inside the phone, nothing leaves it. All opt-in.
Local-first, cloud-optional.The server and dashboard run on your machine. With a local or on-device brain, your screenshots never leave your network. Pick a cloud brain and prompts go to that provider, same as any tool. Your call, always.
- Python 3.10+
- Android: Android phone with USB debugging enabled andADBon PATH (adb devicesshould list your phone)
- iOS: macOS with Xcode, Appium 2 + XCUITest driver, and a trusted iPhone or booted simulator
- Node.js 18+(for the frontend dev server)
Ghost drives iPhones through Appium/WebDriverAgent with the same tool surface:ios:<udid>device refs route tap/swipe/type/screenshot to WDA, and iOS-aware browser primitives (open_url,read_news,extract_visible_text) cover web tasks. Android-only tools (shell,launch_intent, Portal overlay) return a clear platform error instead of failing silently.
iOS isopt-in: enable it withGITD_ENABLE_IOS=1(orios_platform_enabled=truein.env). Wireless drive rides your Tailscale tailnet, so treat the tailnet as the trust boundary. Seedocs/SETUP_IOS.mdfor Appium/WDA setup, andon-device on iPhoneto run the model on the iPhone itself.
doctorprints a checklist with fix hints instead of a stack trace when something is missing (e.g.adbnot on PATH). Preferpipx?pipx install ghost-in-the-droidgives you theghost-in-the-droid/android-agentcommands.
If you have aClaude Max/Prosubscription, you don't need an API key.android-agent loginsigns you in through theclaudeCLI's own Anthropic OAuth flow and points Ghost at theclaude-codeprovider:
android-agent login # opens Anthropic sign-in via the claude CLI
Ghost never handles or stores your token; theclaudeCLI owns it, refresh included.doctorshows a greenClaude subscriptioncheck once you are signed in. To use an API key instead, setANTHROPIC_API_KEY(orOPENAI_API_KEY/OPENROUTER_API_KEY) and pick that provider.
Select Ollama in the Phone Agent tab. Runs entirely on your machine withOllama:
brew install ollama # or: curl -fsSL https://ollama.com/install.sh | sh ollama serve & ollama pull llama3.2:3b # 2GB, fast, good tool-use
git clone https://github.com/ghost-in-the-droid/android-agent.git cd android-agent pip install -e ".[all]" android-agent doctor # preflight android-agent up # start server + dashboard on :5055 # Frontend (separate terminal) cd frontend && npm install && npx vite --host 0.0.0.0 --port 6175 # Dashboard at http://localhost:6175
iOS support requires a Mac because Appium uses Xcode's XCUITest/WebDriverAgent stack. Real iPhones also require trust, Developer Mode, UI Automation permission when prompted, and WDA signing with an Apple development team.
# 1. Install and run Appium XCUITest npm install -g appium appium driver install xcuitest appium --base-path / # 2. Find your iPhone or booted simulator UDID xcrun xctrace list devices xcrun simctl list devices booted # 3. Configure the backend for iOS export IOS_DEVICE_UDID="<udid>" export IOS_APPIUM_URL="http://127.0.0.1:4723" export IOS_BUNDLE_ID="com.google.chrome.ios" # or com.apple.mobilesafari export IOS_MJPEG_SERVER_PORT="9100" # use unique ports per iOS device # 4. Run a product-path smoke workflow uv run python scripts/ios_chrome_news_smoke.py \ --device "ios:<udid>" \ --bundle-id "$IOS_BUNDLE_ID" \ --url https://text.npr.org/ \ --max-headlines 5 \ --max-articles 3 \ --fix-health \ --out-dir data/ios_chrome_news_smoke
For full real-device signing, simulator, WDA MJPEG, health recovery, scheduler, and MCP setup details, seedocs/SETUP_IOS.md.
Copy.env.exampleto.env(if provided) or create a.envfile in the project root. The server reads configuration via Pydantic Settings. Optional variables include:
# Claude Code (same shape for Codex, Cursor, VS Code Copilot, Windsurf) claude mcp add android-agent -- uvx --from ghost-in-the-droid android-agent-mcp
uvxinstalls the package, builds an isolated env, and runs the server. No clone, no venv.
# Codex (OpenAI) codex mcp add android-agent -- uvx --from ghost-in-the-droid android-agent-mcp
Claude Desktop(claude_desktop_config.json),Cursor(.cursor/mcp.json),Windsurf(mcp_config.json):
{ "mcpServers": { "android-agent": { "command": "uvx", "args": ["--from", "ghost-in-the-droid", "android-agent-mcp"] } } }
VS Code Copilot(.vscode/mcp.json) uses the same block under a"servers"key. Contributors who clone the repo get a ready.mcp.json, all 62 tools live on firstclaudelaunch.
toggle_overlay,launch_intent, Android shell helpers, wireless ADB, and Play Store helpers remain Android-only and return stable platform errors on iOS.
New in 1.3: the model can liveon the device. The Ghost companion app embeds real inference engines, so the phone is both the body and the brain. Airplane mode on, agent still working, nothing leaving the device.
On iPhone, Qwen2.5 1.5B runs through llama.cpp on Metal and drives the phone's own UI, with an opt-in MLX engine for faster decode on Apple Silicon. Small models are kept honest with grammar-constrained decoding so tool calls always parse. Full detail on theon-device LLM page.
android-agent skill search tiktok # search the public registry android-agent skill install tiktok # install a skill android-agent skill install github.com/someone/their-skill android-agent skill list # what is installed android-agent skill update tiktok # update a skill android-agent skill validate ./my-skill/ # check before publishing
The skill registry lives inregistry/in this repo. Community skills are auto-discovered nightly from repos taggedandroid-agent-skill.
Community skill(your own repo): create a repo withskill.yaml,elements.yaml, actions, and workflows, tag itandroid-agent-skill, and it appears on theSkill Hubautomatically (nightly scraper).
Official skill(PR to this repo): build and test as a community skill first, then open a PR adding it toregistry/. CI validates, a maintainer reviews, and it gets the "Official" badge.
- skill.yaml: metadata (name, version, app package or iOS bundle ID, supported platforms, actions, workflows)
- elements.yaml: Android UI element resource IDs and descriptions
- elements_ios.yaml: optional iOS selectors for XCTest accessibility trees
- actions/: Python classes extendingActionwithprecondition()andexecute()
- workflows/: Python classes extendingWorkflowwithsteps()
Prefer to let AI do it? TheSkill Creatorwatches a live device stream and generates the action and workflow code while you narrate. Full guide inCONTRIBUTING.md.
A local-first control room athttp://localhost:5055(or:6175in dev). WebRTC and MJPEG live streams, tap-on-screen, tracing, and every tab below.
android-agent/ run.py # Uvicorn entry point on :5055 gitd/ app.py # FastAPI app factory + plugin hook models/ schemas/ # SQLAlchemy 2.0 ORM + Pydantic v2 routers/ services/ # Route handlers + business logic skills/ # Skill packages (tiktok, play_store, safari [iOS], tiktok_ios) bots/common/adb.py # Android Device class: tap, swipe, dump tree, wait_for bots/common/ios.py # iOS Device class: Appium/WDA session, UI tree, gestures mcp_server.py # MCP server: 62 tools for any LLM agent frontend/ # Vue 3 + Vite + TypeScript + Tailwind portal/ # Kotlin companion app (WebRTC, on-device inference) site/ # Docs site (Astro + Starlight)
The flow:FastAPI backend on:5055exposes device control, skills, bots, scheduling, and streaming. The Vue SPA talks to it over/api/*. Skills define per-app interaction (elements + actions + workflows). Device backends route by ref: bare serials usebots/common/adb.py,ios:<udid>refs usebots/common/ios.pyand Appium/WDA. State lives in SQLite via SQLAlchemy 2.0 and Alembic.
Early result, with the full methodology writeup still cooking: driven by Claude Code, Ghost completes115 of 116 tasks (99.1%)onAndroidWorld, Google Research's benchmark for Android agents, on the unmodified upstream harness. Treat this as a preview rather than a citable number. The detailed writeup and trajectories are coming. Watch thereleases.
Most tests are unit/API tests and run without a live device. Live Android and iOS integration tests require the relevant device stack.
# Run all tests on a specific device DEVICE=<serial> python3 -m pytest tests/ -v --tb=short
IOS_LIVE_NEWS_TEST=1 \ IOS_DEVICE_UDID="<udid>" \ IOS_APPIUM_URL="http://127.0.0.1:4723" \ IOS_BUNDLE_ID="com.google.chrome.ios" \ uv run --extra test python -m pytest tests/test_browser_news.py::test_live_ios_chrome_news_workflow
Get your Android device serial fromadb devices.
The project uses Alembic for schema migrations:
alembic revision --autogenerate -m "add new_field to my_table" # after editing a model alembic upgrade head # apply pending alembic downgrade -1 # rollback one
The ghost gets stronger with every skill. SeeCONTRIBUTING.mdfor adding app skills (highest impact), writing actions and workflows, backend architecture, and the PR process. Join the community on theSkill Hub.
MIT. The ghost is free. The ghost is open source. The ghost is yours.
Extentos is a multi-vendor development platform for adding smart-glasses capabilities to existing iOS and Android apps. The simplest analogy is Stripe for smart glasses
An MCP server tailored for React Native–first development using Gluestack UI
MCP Server for PGYER platform, supports uploading, querying apps, etc.
A CLI tool for developers to manage Android devices via ADB.
Drives an Android emulator or a real device over adb: screenshots, UI hierarchy with true device-pixel coordinates, tap and type, app lifecycle, logcat, and Gradle builds and tests.
Interact with Android devices using the Android Debug Bridge (ADB).
Interact with Android devices using the Android Debug Bridge (ADB).
AI 測試大師 — MCP server driving pytest / Jest / Cypress / Go / Maestro. Analyze, generate, run, advise. Web + Mobile (iOS/Android/BlueStacks).
Control Android devices using the Android Debug Bridge (ADB).
Securely browse and read files within an Android project, with built-in validation and access controls for sensitive directories.
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
