agent-desktop-linux

The observe-act loop, refs, JSON envelope, sessions and tracing match every
platform. This file covers only what differs on Linux.

Needs Ubuntu 24.04+ (x86_64 or aarch64), a graphical session of the signed-in
user and a running AT-SPI bus. Other distributions need the at-spi2-core bridge
fix dc0dc33; Ubuntu 22.04's bridge aborts on Action.GetActions. The archive
loader floor is glibc 2.35. Run the CLI inside the target apps' desktop session:
the session bus, display and runtime directory come from its environment. A TTY
login, an SSH shell without session variables, a container without a bus or
another user's session sees no apps. GNOME is the measured desktop; X11 and
Wayland sessions both work and differ in input, capture and window control.
Observation and semantic actions need no helper tool.

Capability Table

First Contact

• Run agent-desktop status first. An empty supported_surfaces means the

accessibility bus did not answer, not an empty desktop. Read
references/bus-and-permissions.md; do not restart a bus you did not start.

• --app takes the name list-apps prints (gnome-calculator,

nautilus) or a desktop-entry name or id backed by verified executable and
argument identity, never a window title or an arbitrary registry label. A
desktop-entry name such as Files is accepted once the entry exists.
list-windows --app is exact (aliases included); list-apps --app combines a
substring of the listed name with exact adapter aliases, ignoring case.

• Window ids are opaque addresses of 34 to 70 characters, valid for the

life of the window's process; run list-windows again after a restart.
list-apps rows carry bundle_id (desktop id when known) and
process_instance.

• GTK4 apps nest deeply. --max-depth and --skeleton count semantic

levels, so empty layout wrappers add output levels. -i --compact still
shows nested nameless group nodes: read past them.

• An empty file view does not prove an empty folder. If an icon view

exposes no file items, try an available list or detailed view, then take a
fresh snapshot and verify which items the application exposes.

• A window appears before its content. Run

wait --window "Text Editor" --app gnome-text-editor first, then
wait --text, which matches a node's own name, value or description. For a
web-engine app (detected from its AT-SPI toolkit metadata), a window
observation waits for three equal full reads and a web-document role, or a
requested depth cut. A shell with no document yet stays incomplete when the
budget ends. A --timeout shorter than the first read returns TIMEOUT,
never a partial tree.

• Geometry-only refs require unchanged geometry. Position and size must

both match, in the ref's recorded coordinate frame; a one-pixel change breaks
identity. After a layout change or a window move, take a fresh snapshot
instead of retrying the old ref. Indistinguishable siblings stay
AMBIGUOUS_TARGET: in a Text Editor window with three tabs, the text views
share one rectangle and one state, so a ref to any of them answers
AMBIGUOUS_TARGET with candidate_count 3 until the extra tabs close. A ref
belongs to the window it was taken in, so two windows with the same title each
resolve their own refs.

• Chromium and Qt apps need a launch flag or variable to appear on the bus;

Electron apps need none, only --no-sandbox where chrome-sandbox is not
setuid. See references/troubleshooting.md.

• Headed ref actions need a window that can take focus. A window the

window manager never activated ends activation_not_granted. See
references/windows-and-lifecycle.md.

• A GTK4 secret field can show its text. Only a password role is withheld.

Treat a GTK4 entry with hidden characters as readable by every observation
command.

• Blocked key combos refuse without --force. See

references/display-servers-and-input.md.

• After opening a menu-role popup, inspect data.surfaces. A menu entry

leads to snapshot --surface menu --app <name> -i. If absent, use
wait --menu and list-surfaces before assuming the opener failed. A
list-backed dropdown is part of the window, not a menu surface; see the shell
reference.

• Shell pointer input requires receiver proof. On X11 a headed shell click

is proven from the shell's own tree: the receiver must be the target and stand
still, or the click refuses before any pointer event. Wayland shell clicks
refuse. Notifications need the GNOME notification service; tray snapshots also
need a StatusNotifierWatcher.

• The state root must be private. A group-writable or world-accessible

AGENT_DESKTOP_HOME or ~/.agent-desktop makes snapshot fail with
INTERNAL; run chmod 700 on it or let the tool create it.

Wait for a window

Wait for an app window to appear (--window requires a title or title substring):

agent-desktop wait --window "Text Editor" --app gnome-text-editor --timeout 10000

To wait for a close that you cause, batch it so the baseline is taken before the
close (Calculator has no unsaved-document prompt to interrupt it):

agent-desktop batch '[{"command":"close-app","args":{"app":"gnome-calculator"}},{"command":"wait","args":{"event":"window-closed","app":"gnome-calculator","timeout":10000}}]' --stop-on-error

To watch someone else close a window, start
agent-desktop wait --event window-closed --app gnome-text-editor --timeout 10000
before they close the window. There is no --window-closed flag.

Cursor overlay

Start a session and pass its returned ID to every overlay command:

session_id=$(agent-desktop session start --name docs-example | jq -r '.data.session_id')
agent-desktop cursor-overlay enable --session "$session_id"
agent-desktop cursor-overlay disable --session "$session_id"
agent-desktop session end "$session_id"

Without a session, cursor-overlay enable and disable return INVALID_ARGS.
On X11, enable returns data.rendered: true; on Wayland it saves the setting
and returns data.rendered: false without drawing an overlay.

session start --cursor --multi-agent returns data.rendered: false, because
nothing is drawn until an agent acts. On X11 each agent id then gets its own
cursor; on Wayland none is drawn.

What differs on Linux

• Firefox/Gecko select can change the selected value without firing change

or input events, so dependent page handlers do not run. The result stays
delivered_unverified even when value readback matches. Check the dependent
application effect before continuing: value readback alone is no commit proof.

• Two click calls on one GTK4 button less than about 250 ms apart act once.

Both answer ok: true with delivered_unverified. To click one button
several times, wait for the effect between the clicks: use
wait --element <ref> --predicate value --value <expected> on the element
that shows the result, or wait 300.

• Ref actions are semantic AT-SPI calls that never move the pointer or change

the active window. Under --headed, ref click, right-click and type try
physical input first and fall back to the semantic action only after a
not_delivered refusal (Windows works the other way round). On Wayland, headed
pointer commands and a headed select refuse at once with
ACTION_NOT_SUPPORTED, cause wayland_session; a headed type needs the
Mutter RemoteDesktop capability. references/display-servers-and-input.md
has the full rules.

• select tries the semantic route first and the pointer second; the pointer

route needs an explicit --headed request on X11. A Qt combo's semantic
selection only highlights its popup row, so headless selection refuses with
POLICY_DENIED, kind pointer_required, before changing that highlight.
Headed, on a list whose toolkit index cannot be mapped to the
row (a GTK4 list) it clicks the row and checks its selected state; on a
combo with no semantic opener (a libadwaita combo row) it clicks the combo
open, selects the choice through the popup's own list and checks the value the
combo shows. Headless, a list refuses with selection_index_unproven, and a
combo with no semantic opener with POLICY_DENIED, kind pointer_required,
unless the combo already shows the choice. A GTK4 drop-down that opens through
its own toggle takes select without --headed and checks the value it
shows.

• A ref can expose no safe semantic mutation route. Headless requests then

return an unsupported-action error without inventing hidden or disabled
state and without switching to physical input. GTK4 entry and password
Activate actions are one such case: use type to edit the field, or request
--headed explicitly when a pointer click is intended.

• Opening a native Wayland Qt combo returns ACTION_NOT_SUPPORTED, kind

native_popup_requires_input, before invoking its action. Qt needs a user
input serial to show that popup. An already open combo may be collapsed;
Qt windows proven to use Xwayland retain their semantic opener.

• Headless type writes through EditableText. A terminal's text node is a

textfield named Terminal whose only action is RightClick; headless
type on it is POLICY_DENIED with check supported_action, and its value is
null. It receives keys only under --headed.

• In key combos cmd and meta are the Super key. Key names are the macOS and

Windows names plus X keysym names. press takes mapped keys only; type
handles characters the layout cannot produce.

• Window ops act on X11 clients; focus-window and headed scoped press require

an X11 session. Headless scoped press requires live target focus on either
display server. Focusing a window on another workspace can switch the user's
current desktop.

• Linux exact FFI refs carrying bounds or a bounds hash require window-relative

geometry, even when the ref also carries a name or identifier. Geometry-free
identity refs are supported.

References

• references/bus-and-permissions.md: bus bootstrap, the three bus states,

status and permissions output, state-root rules.

• references/display-servers-and-input.md: X11 versus Wayland table,

press, type, pointer, wheel, drag, blocked combos, key names.

• references/windows-and-lifecycle.md: launch, close-app, window ops,

focus, workspaces, stopped apps, background press.

• references/capture-and-clipboard.md: screenshot on X11 and Wayland,

the Wayland consent dialog, clipboard owner and helper.

• references/shell-tray-and-notifications.md: menus and GTK4 popovers, GNOME

Overview/top bar, shell reads and input, notification identity guards, SNI
and AppIndicator requirements and refusals, sandbox guidance.

• references/troubleshooting.md: symptom-to-cause map, Chromium, Electron

and Qt flags, toolkit limits, error codes with platform_detail.

Read any file from the binary:
agent-desktop skills get agent-desktop-linux references/troubleshooting.md.