UI automation

The UI tools let your AI operate desktop apps on a node. It reads a window’s controls with ui_tree, acts on them by id with ui_action, and reads the tree again to check the result. click, scroll, type_text and press_keys send real mouse and keyboard input for what the tree doesn’t reach, such as canvases and games. Positions are display pixels, the same as in screenshots.

• Works on Windows, macOS and Linux nodes.
• Like screenshots, it needs the agent in the logged-in desktop session, unlocked and connected.
• Windows: programs running as administrator, and the UAC prompt, can’t be read or controlled unless jump-agent also runs as administrator. The error says so.
• macOS: turn on jump-agent in System Settings > Privacy & Security > Accessibility. The first UI call shows macOS’s prompt for it; until then calls fail with an error that says where to go. If you started jump-agent from Terminal, macOS asks about Terminal instead, and a replaced unsigned build needs the switch turned on again. The front app’s menu bar appears in the window list, so your AI can use menus. Shortcuts use cmd (cmd+s).
• Linux: an X11 session gets everything. On Wayland (the default on recent Ubuntu and Fedora), ui_tree and ui_action work but elements have no positions, and click, scroll, type_text, press_keys and screenshots are refused; choose an X11 session such as “Ubuntu on Xorg” for those. It needs the desktop’s accessibility service, AT-SPI (at-spi2-core, installed with GNOME, KDE, Xfce and most other desktops). GTK, Qt, Firefox, Chromium and LibreOffice apps show their controls. Jump turns the desktop’s accessibility setting on the first time it reads a tree; an app started before that may need restarting to show its controls. Electron apps need --force-renderer-accessibility. Input needs libXtst, present on X11 desktops.
• Input goes to whatever is under the pointer or has focus, so the AI and a person using the node at the same time will get in each other’s way.