Browser Automation

Per-action browser tools over Playwright CLI / Chrome DevTools MCP for navigation, interaction, and data extraction

Overview

Aleph's browser automation is exposed to the agent as a set of focused, single-responsibility tools — not a single browser tool with an action discriminator. Each tool wraps one action (click, navigate, screenshot, snapshot, fill, press a key, upload, hover, scroll, evaluate JS, …) and shares the browser instance through Arc<ProfileManager>.

Two backends are supported underneath:

Playwright CLI (src/browser/playwright_cli_backend.rs)
Chrome DevTools MCP (src/browser/chrome_mcp_backend.rs)

Both surface through a unified BrowserBackend trait — the browser_* tools are backend-agnostic; a profile picks which one starts.

Source locations:

Backends: src/browser/backend.rs
Per-action tools: src/builtin_tools/browser_tools/ (one file per action)
Profile management: src/browser/profile.rs, src/browser/manager.rs
Network policy: src/browser/network_policy.rs (SSRF guard)
Tab registry: src/browser/tab_registry.rs
Chromium discovery: src/browser/discovery.rs

Architecture

┌────────────────────────────────────────────────────────────────────┐
│                  builtin_tools/browser_tools/*                      │
│                                                                     │
│   browser_open → browser_navigate → browser_snapshot                │
│          ↓              ↓                  ↓                         │
│   browser_click / browser_type / browser_fill_form / browser_hover  │
│   browser_scroll / browser_drag / browser_upload / browser_press_key│
│   browser_screenshot / browser_evaluate / browser_console            │
│   browser_dialog / browser_emulate / browser_resize / browser_wait_for│
│   browser_pdf / browser_network / browser_cookies / browser_tabs    │
└──────────────────────────────┬─────────────────────────────────────┘
                               │ Arc<ProfileManager>
                               ▼
┌────────────────────────────────────────────────────────────────────┐
│                       ProfileManager                                │
│                                                                     │
│   Profile { id, backend: PlaywrightCliBackend | ChromeMcpBackend,  │
│             headless, user_data_dir, … }                             │
└──────────────────────────────┬─────────────────────────────────────┘
                               │ BrowserBackend trait
              ┌────────────────┴───────────────────┐
              ▼                                    ▼
┌──────────────────────────┐         ┌──────────────────────────────┐
│  PlaywrightCliBackend    │         │  ChromeMcpBackend             │
│  (CLI subprocess)        │         │  (Chrome DevTools MCP server) │
└──────────────────────────┘         └──────────────────────────────┘

Browser backends

Each profile binds to one backend at creation:

Backend	When to use
`PlaywrightCliBackend`	Default; broadest coverage, no external service.
`ChromeMcpBackend`	When you already run a Chrome DevTools MCP server or want fine-grained CDP control.

browser_profile creates / lists / deletes profiles; browser_session queries the currently active session.

Element targeting

Three targeting methods, by priority:

1. ARIA `ref_id` (preferred)

Every element in an ARIA snapshot has a unique ref_id. This is the most reliable targeting because it references the element through its position in the accessibility tree:

{ "action": "click", "tab_id": "...", "ref_id": "e42" }

Take a browser_snapshot first to obtain the ref_id.

2. Coordinates `(x, y)`

Fallback when ref_id is unavailable — pass viewport coordinates directly.

{ "action": "click", "tab_id": "...", "x": 312, "y": 540 }

3. CSS selector — removed

browser_click (and the other targeting tools) no longer accept a selector field. Passing selector now returns:

CSS selector targeting is no longer supported. Use 'ref_id' from browser_snapshot.

ARIA ref_id and coordinates are both stable and universal.

Tool catalog

Lifecycle

Tool	Description
`browser_profile`	Create / list / delete profiles.
`browser_session`	Query the currently active session.
`browser_open`	Start the profile's browser instance and open a new tab.
`browser_tabs`	List / open / close tabs.

Tool	Description
`browser_navigate`	Navigate an existing tab to a URL.
`browser_snapshot`	Read the ARIA accessibility tree.
`browser_screenshot`	Capture screenshot (PNG / JPEG / WebP), with optional `full_page`.
`browser_evaluate`	Execute JavaScript in the tab.

Element interaction

Tool	Description
`browser_click`	Click an element (`ref_id` or coordinates; `double=true` for double-click).
`browser_type`	Append text (appends to the value, fires an `input` event).
`browser_fill_form`	Fill multiple fields in one structured JSON call.
`browser_press_key`	Press a single key (`Enter` / `Tab` / arrows / etc.).
`browser_hover`	Fire `mouseenter` / `mouseover`.
`browser_scroll`	Scroll (direction + delta).
`browser_select`	Pick an option on a `<select>`.
`browser_drag`	Drag (`from` + `to`).
`browser_upload`	Upload a file through a file input.

Observation and debugging

Tool	Description
`browser_console`	Read console logs.
`browser_network`	Read network requests / responses.
`browser_cookies`	List / set / clear cookies.
`browser_dialog`	Accept / dismiss browser dialogs (alert / confirm / prompt).

Session state

Tool	Description
`browser_emulate`	Emulate device / network conditions.
`browser_resize`	Viewport size.
`browser_wait_for`	Wait for selector / text / function return.
`browser_pdf`	Save the tab as PDF.

Typical workflow

browser_open → browser_snapshot → browser_click / browser_fill_form
                                  → browser_screenshot (optional verify)
                                  → browser_evaluate (optional extraction)
                                  → browser_close_tab (or end browser_session)

Every tool takes a profile field (default "default") plus backend-specific extras.

ARIA accessibility snapshot

Snapshots give a structured view of the page. browser_snapshot returns:

{
  "elements": [
    {
      "ref_id": "e1",
      "role": "heading",
      "name": "Welcome",
      "bounds": { "x": 100, "y": 50, "width": 300, "height": 40 }
    },
    {
      "ref_id": "e42",
      "role": "button",
      "name": "Submit",
      "state": ["focused"],
      "bounds": { "x": 200, "y": 400, "width": 120, "height": 36 }
    }
  ],
  "page_title": "My Page",
  "page_url": "https://example.com",
  "focused_ref": "e42"
}

ref_id is valid for one snapshot only; the next browser_snapshot reassigns them.

Approval policy

All browser_* tools accept a user-defined ApprovalPolicy (src/approval/) that gates sensitive actions:

Action category	Gated
`browser_open`, `browser_profile`, `browser_session`, `browser_close_tab`	No (lifecycle)
`browser_snapshot`, `browser_screenshot`, `browser_console`, `browser_network`, `browser_cookies`	No (read-only)
`browser_navigate`, `browser_click`, `browser_type`, `browser_fill_form`, `browser_press_key`, `browser_hover`, `browser_scroll`, `browser_select`, `browser_drag`, `browser_upload`, `browser_evaluate`, `browser_dialog`, `browser_emulate`, `browser_resize`, `browser_pdf`	Yes (navigation / mutation / input)

When the policy denies, the call returns success: false with the reason; when it requires confirmation, it returns success: false with Approval required: … — the agent relays that to the user.

Network policy

All browser navigation goes through BrowserSsrfGuard (src/browser/network_policy.rs) — sharing SsrfPolicy with the tool SSRF engine. Private network, link-local, loopback, and cloud-metadata endpoints are rejected by default.

Graceful degradation

browser_* calls against an unconfigured profile return a friendly message instead of an error:

{
  "success": false,
  "message": "No browser profile configured. Use browser_profile to create one first."
}

The agent can create a profile and start a browser without error-handling boilerplate.

Security notes

Process isolation: the browser runs as a separate Chromium process inside its own sandbox.
Headless mode: use headless: true in production to avoid visual interference.
Approval policy: gate sensitive actions (navigation, JS evaluation, upload) through the approval system.
User data directory: use a dedicated browser profile to isolate cookies and storage from the user's personal browser.
JS evaluation: browser_evaluate can execute arbitrary JS in the page context — powerful but should be gated by an approval policy in security-sensitive deployments.
Anti-automation flags: default Chromium flags reduce bot detection, but some sites may still block automated access.

Example: search and extract

1. browser_profile(action="create", id="default", headless=true)
   → profile created

2. browser_open(profile="default", url="https://blog.rust-lang.org/")
   → tab_id="TAB_001"

3. browser_snapshot(profile="default", tab_id="TAB_001")
   → { elements: [
         { ref_id: "e1", role: "heading", name: "Rust Blog" },
         { ref_id: "e5", role: "link", name: "Rust 1.84.0" },
         …
       ]}

4. browser_click(profile="default", tab_id="TAB_001", ref_id="e5")
   → success

5. browser_snapshot(profile="default", tab_id="TAB_001")
   → new ref ids for the announcement page

6. browser_evaluate(profile="default", tab_id="TAB_001",
                    js="document.body.innerText")
   → extracted text

7. browser_tabs(action="close", tab_id="TAB_001")