GitHub · by Lahfir

Agent Desktop

Desktop control from the accessibility tree, with Jev picking the operation.

Lahfir’s Rust tool gives an agent reliable computer use on a real desktop. It reads an application’s own UI structure through the operating system’s accessibility tree rather than guessing from pixels, so element references stay stable and actions are safe to retry. The repository ships a `jev-desktop` skill that drives the loop with Jev.

The skill has two entry points over the same screen reading: hand it a whole goal and it observes, decides and acts until the goal is met, or hand it one step and keep the plan yourself. Neither puts the tree in the calling agent’s context, which is the point — a window holding 150 elements would otherwise cost that many lines of JSON on every turn.

Each turn asks for one operation and a target for that operation in the same request. The operations are CLICK, TYPE_TEXT, CHECK, UNCHECK, EXPAND, COLLAPSE, SCROLL, DRILL, WIDEN, WAIT, DONE and BLOCKED, and only the ones something on screen can receive are offered. The target is drawn from the elements that advertise the matching action, so a target can never be incompatible with the verb acting on it. That is why the loop does not need to correct itself afterwards.

Looking is itself an operation. A window is read as a skeleton first, so a document holding four thousand elements costs the same first look as a panel holding thirty; DRILL pins a cut-off region as the root for later turns and WIDEN gives the whole window back. CHECK and UNCHECK are used instead of a toggle because both are idempotent, so a box already in the wanted state stays there and nothing has to reason about the current one. The run never invents a value to type: when no text is left to give, TYPE_TEXT is not offered at all.