Browser & Desktop Control

AGENTOS DOCS / AUTOMATION & INTEGRATION

With Local UI Control on, your agent can use your computer the way you do: browse in your logged-in sessions, click and type, read native app windows and take screenshots.

// OPT-IN Every Local UI feature is off until you turn it on in Global Settings → Local UI Control. When it's on, anything visible in the controlled browser or on screen may be sent to your model. Use supervised mode while you get comfortable.

Browser control

Choose one of three modes:

Mode	Best for	Setup
Debug port	Fastest; uses a browser you launch	Start Chrome or Edge with `--remote-debugging-port=9222`
Chrome extension	Always-on access to your normal browser, even after restarts	Install the AgentOS CDP Bridge extension
Playwright	Fully automated; AgentOS launches its own browser	None. Chromium is bundled.

Debug port

"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222

Then set Browser Control to Enabled, Mode to debug, and Debug Port to 9222.

Chrome extension (CDP Bridge)

Download the AgentOS CDP Bridge extension, open chrome://extensions, turn on Developer mode, click Load unpacked and select the extension folder.
In the extension's options, set the WebSocket URL to ws://localhost:3456/__cdp_bridge__ (use your port if you changed it).
Set Browser Control to Enabled and Mode to extension.

When it connects, the server log shows CDP Bridge: Chrome extension connected.

Playwright

Set Mode to playwright and choose Chromium, Firefox or WebKit, headed or headless. If the bundled browser is missing, AgentOS falls back to an installed Chrome or Edge.

Keyboard & mouse input

Turn on Input Simulation to let the agent move the mouse, click, type, press shortcuts (such as Ctrl+C or Alt+Tab) and scroll in any window. This works on Windows with no extra software.

Accessibility control

Turn on Accessibility Control to let the agent read and operate native app windows by name (buttons, fields, menus, checkboxes) without needing pixel coordinates.

Platform	Support
Windows	Built in (UI Automation)
macOS	Built in. Grant access in System Settings → Privacy & Security → Accessibility.
Linux	Not yet supported. Use input simulation instead.

Screenshots & vision

The agent can capture the full screen, a browser page's content, or the accessibility tree. With a vision-capable model, it can also look at the screen and decide where to click. Screenshots are saved temporarily to ~/.agentos/tmp/.

Example prompts

Connect to my browser and fill out the contact form at https://example.com/contact,
then click Submit.
Take a screenshot and tell me which application is in focus.
In the Settings window that's open, set the "Username" field to "admin".

How AgentOS picks a method

For UI tasks, AgentOS tries the browser first (if one is connected), then accessibility control, then input simulation. It retries each method (2 times by default) before moving on to the next.