Computer Use AI SDK
- agent-framework
About
What is Computer Use AI SDK?
Computer Use AI SDK is an open-source MCP server and client that enables AI agents to control a computer natively on macOS. It uses underlying desktop-rendered elements rather than pixel-based vision, making it faster and more reliable than approaches like OpenAI’s Operator or Claude’s computer use.
How to use Computer Use AI SDK?
Clone the repository, install Rust and Node.js dependencies, then run the backend server (cargo run --bin server) and choose either the CLI interface (cd mcp-client-cli-interface, npm install, set ANTHROPIC_API_KEY, npx tsx main.ts) or the web app interface (cd mcp-client-nextjs, npm install, set API key in .env, npm run dev).
Key features of Computer Use AI SDK
- Native macOS support without virtual machines or guardrails
- Relies on desktop-rendered elements, not pixel-based vision
- Provides tools: launch apps, read content, click, enter text, press keys
- Two interfaces: CLI for debugging and Next.js web app
- Open source with full customizability
Use cases of Computer Use AI SDK
- Build custom agent workflows to automate repetitive desktop tasks
- Create a custom UI for end‑users to control computer actions
- Save workflows and run them on a cron schedule
- Combine with other MCP servers, e.g., fill a Google Sheet from daily chat history
FAQ from Computer Use AI SDK
What does Computer Use AI SDK do?
It allows AI agents to control a computer by providing tools like launching apps, reading content, clicking, entering text, and pressing keys—all using native desktop elements instead of pixel screenshots.
What platforms does it support?
It is native to macOS only; the README states “no virtual machine bs, no guardrails” and demonstrates only macOS use.
Does it require an API key?
Yes, it requires an Anthropic API key (set as ANTHROPIC_API_KEY) to power the AI agent.
Is it open source?
Yes, the repository is publicly available with an MIT‑style license (implied by the GitHub repo and “open source alternative” statement).
How does it differ from pixel‑based vision models?
It uses underlying desktop‑rendered elements rather than analyzing pixels, making it much faster and far more reliable than pixel‑based vision models.
Details
- Author
- mediar-ai
- GitHub stars
- 160
- Category
- agent-framework
- Repository
- mediar-ai/mcp-server-client-computer-use-ai-sdk
Computer Use AI SDK
We've built an MCP server that controls computer
You've heard of OpenAI's operator, you've heard of Claude's computer use. Now the open source alternative: Computer Use SDK from screenpipe.
It's native on macOS—no virtual machine bs, no guardrails. Use it with any app or website however you want.
No pixel-based bs—it relies on underlying desktop-rendered elements, making it much faster and far more reliable than pixel-based vision models.
You can now build your own agents getting started with our simple Hello World Template using our MCP server and client.
There are tools that our MCP Server provides out of the box:
Launch apps
Read content
Click
Enter text
Press keys
These will be computational primitives to allow the AI to control your computer and do your tasks for you. What will you build? Come check us out at https://screenpi.pe
Demos
agent sending a message
https://github.com/user-attachments/assets/f8687500-9a8c-4a96-81b6-77562feff093
get latest whatsapp messages
open arc browser
Get started
git clone https://github.com/m13v/computer-use-ai-sdk.git
cd MCP-server-client-computer-use-ai-sdk
# Install Rust (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Install Node.js and npm (if not already installed)
Visit https://nodejs.org/ or use nvm
# run backend server
cd mcp-server-os-level
cargo run --bin server
keep it running
Option 1: CLI Interface
# run CLI interface client in a new terminal (good for debugging)
cd mcp-client-cli-interface
npm install # install dependencies first
Set your Anthropic API key as an environment variable
export ANTHROPIC_API_KEY=sk-ant-xxxx # Replace with your actual Anthropic API key
For Windows, use: set ANTHROPIC_API_KEY=sk-ant-xxxx
For permanent setup, add to your shell profile (.bashrc, .zshrc, etc.)
npx tsx main.ts
Option 2: Web app Interface
# run CLI interface client in a new terminal (good for debugging)
cd mcp-client-nextjs
npm install # install dependencies first
Set API key via command line
echo "ANTHROPIC_API_KEY=sk-ant-XXXXXXXX" > .env # replace XXXXXXXX with your actual key
Or append if you want to keep other env variables
echo "ANTHROPIC_API_KEY=sk-ant-XXXXXXXX" >> .env
npm run dev
go to provided localhost web page
What do I do with it?
- Build custom worfklows of agents to performs various actions
- Build custom UI to make it easy for users to automate their computer work
- Save workflow and run in cron
- Combine with other MCP servers to do something cool, e.g.: fill out a google sheet based on the history of people i talk to throughout the day
Request features and endpoints in github issues
https://github.com/m13v/computer-use-ai-sdk/issues/new/choose
