Desktop Bridge

The Desktop Bridge gives Aleph physical-world capabilities: taking screenshots, reading screen text via OCR, clicking buttons, typing text, managing windows, and rendering canvas overlays. It is the primary implementation of the "Muscles" limb in the 1-2-3-4 model.

Architecture

The bridge follows the Brain-Limb separation principle (redline R1): the Rust Core never calls platform-specific APIs directly. Instead, it spawns a long-lived aleph-bridge Swift child process (via tokio::process::Command) that links AVFoundation, Vision, Accessibility, and CoreGraphics, and exchanges JSON-RPC 2.0 requests on the child's stdin/stdout. Stderr is forwarded to the Rust tracing subscriber.

┌──────────────────────┐       stdio (line-delimited JSON-RPC 2.0)
│     aleph-server     │  ──stdin─────────────────────────────▶   ┌──────────────────────┐
│     (Brain)          │                                            │   aleph-bridge       │
│                      │  ◀──stdout────────────────────────────────│   (Swift helper)     │
│  SwiftBridge client  │                                            │                      │
│  (desktop/shared/    │  stderr── tracing logs ──────────────────▶│  AppKit / Vision /   │
│   src/bridge/)       │                                            │  CoreGraphics / ...  │
└──────────────────────┘                                            └──────────────────────┘
        Rust Core                                                       Native Platform

The bridge runs headless on every desktop OS that Aleph targets; the helper process exposes its capabilities through the JSON-RPC surface, never through a UI.

This separation means:

• The Core can run on a headless server with no desktop capabilities
• Desktop capabilities can be upgraded independently of the Core
• Platform-specific code is isolated in the Swift helper, not in src/builtin_tools/desktop/

Spawn lifecycle

The SwiftBridge client (desktop/shared/src/bridge/client.rs) spawns the helper lazily on the first call. A Supervisor (desktop/shared/src/bridge/supervisor.rs) watches the child via stdout EOF — when the helper closes stdout the supervisor drains the inflight table, resets the state slot, and records the crash in the shared SpawnGate. The respawn ladder is 1s → 2s → 4s → 8s → 16s, capped at 30 s.

After five crashes within a ten-minute window the bridge enters disabled mode: all subsequent calls return DesktopError::BridgeDisabled immediately and no further respawns are attempted until the server restarts. A spawn request that arrives inside the backoff window (but below the disable threshold) returns DesktopError::BridgeBackoff without spawning — pacing the restarts so a crash/spawn-failure loop never burns the window in milliseconds.

The helper installs a parent-death watchdog: it polls getppid() and exits cleanly if the parent PID changes, preventing zombie helper processes when the server crashes.

Per-call RPC timeout

Crash recovery only fires when the helper closes stdout (EOF). A helper that accepts a request and then hangs — stuck in a native API, deadlocked — keeps stdout open, so the reader loop never observes EOF. To stop such a helper from wedging an agent turn indefinitely, every RPC is bounded by a per-call deadline:

• SwiftBridge::call spends the deadline the protocol declares for that method (methods::suggested_timeout_ms → bridge::client::rpc_timeout_for). Resolution is: an exact per-method override, else the namespace default, else the client's DEFAULT_RPC_TIMEOUT (60 s) for a method outside every known namespace.
• SwiftBridge::call_with_timeout takes an explicit deadline, and is for operations whose length is a function of their arguments: camera.clip and audio.record pass requested_duration + 30 s; speech.transcribe_file passes a flat 300 s.

Deadlines live next to the method constants they belong to (shared/protocol/src/desktop_bridge/methods/*.rs, DEFAULT_TIMEOUT_MS + TIMEOUT_OVERRIDES_MS). Current values:

On timeout the caller receives DesktopError::BridgeTimeout, the in-flight slot is dropped (no leak), and the helper is left running — only that one call fails. A late reply from a merely-slow helper is discarded by the reader.

Protocol

Every interaction follows JSON-RPC 2.0 over a line-delimited stdio stream. Each request writes one JSON line to the child's stdin; each response is one JSON line on stdout. The protocol schemas live in shared/protocol/src/desktop_bridge/:

• envelope.rs — Message / Request / RpcError
• errors.rs — wire codes (ERR_TIMEOUT, ERR_HELPER_CRASHED, ERR_NOT_IMPLEMENTED, ERR_PERMISSION_DENIED, ERR_BRIDGE_DISABLED, ERR_PLATFORM)
• methods/{ax,bridge,input,media,perm,pim,screen}.rs — per-namespace schemas and deadlines

Request format

{
    "jsonrpc": "2.0",
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "method": "screen.capture",
    "params": {
        "region": { "x": 0, "y": 0, "width": 1920, "height": 1080 }
    }
}

Response format

{
    "jsonrpc": "2.0",
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "result": {
        "png_base64": "iVBORw0KGgo...",
        "width": 2560,
        "height": 1600
    }
}

Error format

{
    "jsonrpc": "2.0",
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "error": {
        "code": -32002,
        "message": "not implemented: pim.mail.send"
    }
}

Server-defined codes (errors.rs) carry semantics the caller needs to distinguish — most importantly -32002 NotImplemented (the method exists, the capability is deliberately absent, e.g. pim.mail.* on macOS) vs -32601 MethodNotFound (a real wiring gap). -32001 PermissionDenied carries a PermissionGuide in data so the LLM can surface the deep link, the human-readable steps, and the rationale.

Capabilities

The bridge exposes capabilities in seven namespaces. The full per-method surface lives in shared/protocol/src/desktop_bridge/methods/; this table summarises the most-used methods grouped by capability area.

screen.* — Perception

ax.* — Accessibility (macOS / Windows UIA / Linux AT-SPI)

input.* — Pointer / keyboard / clipboard

window.* — Window / app management

window.list, window.focus, window.move, window.resize, window.launch, window.quit, window.restart. macOS resolves a CGWindowID to its AXUIElement window by geometry (desktop/shared/src/action/window_ax.rs); osascript is the fallback when AX refuses.

media.* — Camera / audio / speech

media.camera.snap, media.camera.clip, media.audio.{list_devices,mic_meter,record,record_stop}, media.speech.transcribe_file.

perm.* — Permission introspection

Read-only — perm.list, perm.status, perm.request. The helper exposes TCC permission state and lets callers request a grant so the OS can pop the system prompt.

pim.* — Contacts / mail / calendar (where the platform has them)

pim.contacts.*, pim.mail.*, pim.calendar.*. macOS returns NotImplemented for these today.

Error Handling

DesktopError (desktop/shared/src/error.rs) is the typed surface the Rust caller sees:

pub enum DesktopError {
    NotAvailable(String),       // capability missing on this platform / config
    ScreenCapture(String),      // capture / crop failed
    InputFailed(String),        // pointer / keyboard failed
    OcrFailed(String),          // OCR processing failed
    WindowFailed(String),       // window management failed
    NotImplemented(String),     // method exists, capability deliberately absent
    BridgeFailed(String),       // helper crashed / unexpected output
    BridgeTimeout(String),      // helper accepted the call, did not reply in time
    BridgeBackoff(String),      // spawn cooling down inside restart-backoff window
    BridgeDisabled(String),     // >5 crashes inside 10-minute restart window
    PlatformError(String),      // underlying OS error (e.g. IOPMAssertion)
    PermissionDenied { kind, guide }, // TCC denied; `guide` carries deep link + steps
}

Server-defined codes are mapped to typed variants inside SwiftBridge (shared/bridge/client.rs::map_rpc_error) so callers see the difference between "the helper is unresponsive" (BridgeTimeout), "the helper is crash-looping" (BridgeBackoff / BridgeDisabled), and "the OS refused" (PermissionDenied).

Source map

26.7.x Addendum

macOS Screen Recording Fixes

26.7.21+ (desktop/shared/src/perception/screen_record.rs):

• Screen capture cropped to the requested region via setSourceRect (physical pixels, not display points; the region is converted through the surface's scale).
• Completion verified to have produced output before reporting success — a zero-byte stream used to be reported as a successful capture.
• Typed bridge errors preserved on the OCR / window-capture / media / input / screen rails instead of being flattened to a generic BridgeFailed.
• screen_record serialization errors propagate instead of becoming null.

macOS Window Targeting

26.7.21+ public AX window resolver (desktop/shared/src/action/window_ax.rs): CGWindowID → AXUIElement by matching the AX window whose position+size equals that window's global bounds (from CGWindowListCopyWindowInfo). AX and CGWindowList both report top-left-origin global points, so the geometry compares directly. No private symbols (_AXUIElementGetWindow) are used. osascript is the fallback when AX refuses — focus_window raises the specific target window, not whichever window happens to share its title.

Per-call RPC Timeouts Are Now Real

Previously the deadline constants existed for years as ten free-floating SUGGESTED_TIMEOUT_MS* with zero consumers — every call rode the 60 s catch-all. The two that hurt were ax.query_focused (the type_text focus gate issues it before every keystroke batch — 3 s intended, 60 s actual) and screen.capture (which has an xcap fallback on macOS, so the deadline was exactly how long a wedged helper delayed a capture that would have succeeded instantly on the other transport). Now every method inherits a sane budget from its namespace, with overrides where they earn their keep.

Spawn pacing closes a burn-through window

Earlier the supervisor computed the backoff delay and then discarded it, leaving the ladder inert — a crash/spawn-failure loop could burn through the 10-minute restart window in milliseconds. ensure_running now asks SpawnGate whether enough time has elapsed and returns BridgeBackoff without spawning during the cool-down.

iOS WKWebView / Keychain

26.7.18+ certificate-trust adapter + iOS Keychain trust store + decision mirror.

Bridge Protocol Stable

The stdio JSON-RPC 2.0 interface is stable. See Architecture 1-2-3-4 Model.

See Also

• Architectural Redlines R1 — platform-bridge constraint
• Browser Automation — desktop tools
• Gateway Protocol TLS — self-signed + TOFU