hypruse

No ydotool daemon. No root. No portals. No X11.

Computer use exists on macOS and Windows. On Linux there is effectively nothing: the Claude Desktop Linux beta explicitly shipswithoutscreen control, Anthropic's reference implementation is an X11 container, and the existing Wayland attempts lean on setuid uinput hacks or GNOME-only portals.

Meanwhile Hyprland already exposes everything an agent needs, better than any accessibility bridge: a complete IPC surface for state and window management, and first-class Wayland protocols for input. hypruse just wires them to MCP:

agent (Claude Code, or any MCP client) │ stdio ▼ hypruse ├── hyprctl -j ········▶ desktop state: monitors, workspaces, windows, layers ├── hyprctl dispatch ··▶ focus / move / close / launch / movecursor ├── grim ··············▶ screenshots: monitor, window crop, region ├── busctl (AT-SPI) ···▶ accessibility tree: named controls, current │ values, exact coords (ui / marks / click_ui) ├── wtype ·············▶ keyboard (zwp_virtual_keyboard_v1, real XKB keymap) └── raw Wayland wire ··▶ click & scroll (zwlr_virtual_pointer_v1)

Optional binaries gate two more tools:imagemagickdraws the numbered overlay formarks, andwl-clipboardbacks the opt-inclipboardtool.

The acting tools (pointer,keyboard,click_ui,hypr,use_bind,sequence) take an optionalthenargument that appends the result to the same call, so the agent sees the effect without a second round-trip:then='desktop'adds a fresh semantic snapshot (~20 ms, cheap, best for window/focus changes),then='screenshot'a stable capture (best for visual changes),then='ui'the acted-on window's controls with their current values (a few hundred exact tokens, best after typing or toggling;click_uireads the window it clicked even if the click handed focus to a dialog, the others read the focused window),then='none'nothing (the default everywhere exceptsequence, which defaults to'desktop').

The tools group into five capabilities, ordered most-reliable-and-cheapest first. An agent that reaches for them in this order is both faster and more accurate, and many tasks never need a screenshot at all.

1. Semantic desktop control (start here)

desktopreturns the entire window and workspace tree in one call: every window's address, class, title, and geometry, the active window, the cursor, and any layer surfaces on screen (launchers, bars, notification popups).hyprandlaunchthen act on it over IPC in milliseconds: switch workspace, focus/move/close/fullscreen/float a window by address, or start an app.

Use it well:never take a screenshot to find or arrange windows. Readdesktop, act on the address you want.launchblocks on the realopenwindowevent and hands back the new window's address, so there is nothing to poll or guess; it also relocates single-instance apps (browsers) that ignore workspace rules.

When an app exposes an accessibility tree (most GTK and Qt apps), you can target controls by name instead of by pixel.uilists every control with its exact global coordinate, and reports the current value of the controls that carry one: the text in a field, a slider's percentage, a checkbox's state.click_uiresolves a name and clicks it in one call, through the real cursor (so the beacon and every safety guarantee still apply).marksdraws numbered marks over a screenshot with a legend, for when you want to see the options first and thenclick_ui(mark=N).

Use it well:reach forclick_ui name="Save"before estimating any pixel, since it is exact and spends no image. Read a form's state withui(did the box actually tick?) instead of screenshotting it. An ambiguous name returns the candidates rather than guessing. When an app exposes no tree (terminals, canvas apps, Electron/Chrome without--force-renderer-accessibility) the tool says so, and you fall back to vision.

3. Vision when it matters: the zoom loop

For everything the accessibility tree cannot name,screenshot(monitor, window crop, or region) andzoom(a native-resolution re-capture around a point) carry a strict coordinate contract,global = geometry + pixel / scale, that stays exact on every monitor and fractional scale.

For an agent the model calls dominate task latency, not the desktop, so the real speedups are structural.sequenceruns an ordered micro-plan (click, type, press enter, wait) in a single call, stopping the moment the desktop changesstructurallyin a way a step did not intend (a window opening, closing, or moving, an unexpected workspace switch, or a seat-taking launcher or on-screen keyboard; it deliberately ignores bare focus changes and notification popups).then='desktop' | 'screenshot' | 'ui'fuses a fresh view of the result into the acting call itself.wait_forblocks on real compositor events (a window or launcher opening, a title changing, a workspace switch, an urgency hint, screen-sharing starting) instead of sleeping and hoping.

Use it well:collapse a known click/type/enter flow into onesequence. After typing into a form, addthen='ui'to read the effect back in a few hundred exact tokens instead of a screenshot. After a launch or a shortcut that opens something,wait_forthe event rather than sleeping.

hypruse hands an agent your real seat, so it ships the controls to bound what that agent can do. Beyond the always-on approval prompts and the Waybar activity beacon, opt-in env flags narrow what an agent can touch:HYPRUSE_READONLYexposes only the observation tools;HYPRUSE_CONFINErestricts input to the windows the agent launched, or a class/workspace allowlist;HYPRUSE_AUTH_GUARD(on by default) refuses to drive authentication dialogs;HYPRUSE_STRICTrefuses to act if you took the seat back;HYPRUSE_MARKtags agent-owned windows and announces when the agent opens a window or captures the screen. Two more record rather than restrict:HYPRUSE_JOURNALwrites an auditable NDJSON line per tool call, refusals included, andHYPRUSE_DRYRUNruns every check and delivers nothing, so you can watch an agent plan the work before it touches your desktop.

yay -S hypruse # or hypruse-git for main

Then let it set itself up and verify the environment:

hypruse init # detects your MCP clients, registers (asks first), runs doctor hypruse doctor # just the diagnostics

claude mcp add -s user hypruse -- uvx hypruse

claude mcp add -s user hypruse -- uv run --directory /path/to/hypruse hypruse

Read-only mode:setHYPRUSE_READONLY=1in the server config to expose only the observation tools (desktop,screenshot,zoom,ui,marks,binds,wait_for). The agent can see and narrate but cannot click, type, or launch. A good first week.

The Linux beta shipswithoutAnthropic's first-party computer use, but stdio MCP servers work in chat, which makes hypruse the workaround. In~/.config/Claude/claude_desktop_config.json:

Six opt-in env flags. The first four narrow what an agent can touch, and each fails towardlessaction; the last two record and rehearse rather than restrict. All compose with the layers above:

The flags above decide what an agent may do in the moment and then forget it happened.HYPRUSE_JOURNALis the memory. One JSON object per line, sotail -f,grep, andjqall work on a live file:

{"v":1,"seq":7,"ts":"2026-08-02T09:12:13.456Z","kind":"act","tool":"pointer", "args":{"action":"click","x":800,"y":60},"outcome":"ok","ms":37,"result":"click ok"}

kindsplits the two questions people actually ask:actis input delivered to your desktop,observeis the agent looking, which is what a privacy audit wants (when the screen was captured, when the clipboard was read). Observationresultsare never recorded, only that they happened, so the journal never becomes a second copy of everything the agent saw.Refusals are recorded too, with the guard's own message: it is the only place the history of your trust layers doing their job exists.

Typed and copied text is recorded as a length plus a short digest, not as text, because keystrokes are passwords.HYPRUSE_JOURNAL_TEXT=1keeps it verbatim, which you need only to replay typing. Note what a digest is and is not: it proves two entries typed the same thing and it will not hand a reader your password, but it is an unsalted SHA-256 prefix next to an exact character count, so a four-digit PIN is trivially recovered from it. Treat the journal as sensitive either way. It is written 0600 in a 0700 directory, and rotated atHYPRUSE_JOURNAL_MAX_BYTES(8 MiB, one previous generation kept as.1; set 0 to never rotate).

The journal is a recorder, not a guard: if it cannot be written the action still happens and hypruse warns once on stderr, because failing your desktop over a log line is the wrong trade. What it doesnotrecord is athen=observation as its own entry: the acting call that carried it is recorded,then=screenshotand all, but the capture does not get a second line of its own.

Read it back withhypruse journal(--actsfor actions only,--refusedfor what the guards stopped,-n Nto tail,-vto include each result):

1 09:12:10 session start pid 4211 0.10.0 confine=launched auth_guard=1 strict=True 14 09:12:13 act pointer action=click x=800 y=60 16 09:12:15 act keyboard action=type text=<11 chars> 19 09:12:19 act hypr action=close_window target=0x5f2a10 REFUSED TrustError: 0x5f2a10 (Signal) is outside the agent's confinement scope 312 actions (0 dry), 604 observations, 1 refused by a trust layer, 0 errors

HYPRUSE_DRYRUN=1turns the same session into a rehearsal. Every acting tool validates its arguments and runs every trust guard, then reports the plan instead of executing it, so a dry run refuses exactly what a real run would:

DRY RUN, nothing was delivered: would click push button 'Send' at (1204, 880) in signal

Nothing reaches the desktop: not the click, not the keystroke, not even the window focus that normally precedes typing. Enforced twice, once at each tool and once at the input path itself, so a code path nobody thought of fails loudly rather than quietly acting during a simulation. The agent is told dry run is on, so it reports a plan instead of retrying an action that "did not work". The scope is the agent's actions, not the server's own startup:HYPRUSE_MARK, if you set it, still installs its window rule when the server starts.

hypruse replay <journal>re-issues a journal's actions through the same tool functions, so the same guards apply to the replay. It prints the plan and stops there unless you pass--execute. Before it takes the seat it refuses outright, rather than failing halfway and leaving your desktop part-way through someone else's plan, when: an action was recorded by a newer hypruse, a recorded window no longer exists (--skip-missingruns the rest), typed text was recorded as a digest, the entry is aclick_ui(mark=N)whose numbering died with the session that drew it, the entry is a clipboard write andHYPRUSE_CLIPBOARDis not set, orHYPRUSE_READONLYorHYPRUSE_DRYRUNis set. It paces itself from the recorded timing, capped by--max-gapand scaled by--speed, and its own actions are recorded and marked, so replaying the same file again runs the original plan rather than the plan plus the replay of it.

The honest limit is window addresses: they are heap pointers, so yesterday's journal mostly names windows that are gone, and an address can even be reused by a different window later, which no pre-flight can catch. Replay is for re-running a flow on a desktop that still looks like the one recorded.

Dry run and replay compose in the obvious direction: let the agent work withHYPRUSE_DRYRUN=1, read the journal, and replay it with--executeonce the plan is one you like.

Measured on a live session (Hyprland 0.55, 1080p, 20 windows):desktop~20 ms (one batchedhyprctlcall), workspace/window dispatch ~10-20 ms, full-monitor screenshot ~65 ms (fast JPEG default; ~800 ms if you ask for lossless PNG, grim's zlib path dominates), region/zoom captures well under that. If tool callsfeelslow, it is almost certainly the MCPapproval promptin front of each call, not the server. Allowlist the tools you trust and the latency disappears. Claude Code (.claude/settings.json):

Captures default to JPEG q90: on a 1080p frame that is roughly 12x faster to encode than PNG (grim's zlib path dominates capture time, measured ~65 ms vs ~800 ms) and 3-4x smaller, while full-res q90 reads UI text well. Passlossless=truefor exact pixels (PNG). In image mode, captures also fit the host's result-size limit (Claude Desktop caps tool results at 1 MB) by degrading quality before resolution, since grim's downscale filter is slower than a full-res capture, and cap the long edge atHYPRUSE_MAX_IMAGE_EDGEpixels (default 1568) so the host never downscales the image under the model. The applied scale is folded into the returned metadata, so coordinate mapping stays exact; tune withHYPRUSE_MAX_IMAGE_BYTES, or passscalefor a deliberate zoom-out.

By default the screenshot tool writes the image under$XDG_RUNTIME_DIR/hypruse/and returns its path; MCP hosts with a file reader (Claude Code'sRead) render it natively. This default exists because some hosts (including Claude Code 2.1.x) serialize inline MCP image blocks to base64 text the model cannot see.HYPRUSE_SCREENSHOT_MODE=imageswitches to inline image content blocks for hosts that render them correctly.

uv sync --group dev uv run pytest # unit tests, no compositor needed uv run pytest -m e2e --override-ini addopts= # live seat-safe checks uv run python scripts/e2e_input.py # supervised: takes the seat ~10s

The input e2e is deliberately manual: it borrows your cursor and keyboard, counts down, proves click/scroll/type delivery by reading the target terminal's screen back over kitty remote control, and restores your focus.

- In-process wlr-screencopy over the raw wire (as input already works): drop grim's fork floor from small captures and add damage-tracked wait-for-stable that returns the instant the screen settles.

- Semantic screen diff: after an action, return only what changed (window topology from the event stream, or a bounded changed-region crop) instead of a full frame. The socket2 event expansion behindwait_foralready tracks most of the topology; the missing piece is folding it into a post-action delta.

- Wider accessibility coverage: close the gaps theuiandmarkstools hit today, chiefly GTK's newer combo boxes that publish neither their text nor selection (so a rendered dropdown value still needs a screenshot), plus AT-SPI value-change events sothen='ui'can report a control settling without a poll.
- Notifications: read recent desktop-notification content and history, not just wait for the popup to appear (wait_foralready matcheslayer_openon the notification namespace).

- sway / niri support: the wire client already speaks the wlr protocols; what remains is an IPC layer alongsidehyprctl.py(contributions welcome).
- Headless end-to-end tests in CI: needs a QEMU virtio-gpu VM, since Hyprland's aquamarine backend requires a real GPU render node that hosted runners lack.

- Zoom-loop precision benchmark: measure click accuracy of the coarse-to-fine loop against known targets.
- End-to-end task-success benchmark on Hyprland (OSWorld-style): the deferred bigger sibling of the zoom-loop microbenchmark, scoring full multi-step tasks through the real MCP surface so the a11y-versus-vision and round-trip work is judged on task completion, not latency alone.