Computer Use & Live Audio

This page covers three related capabilities: the provider-agnostic computer use (CU) abstraction that lets a model see and drive a desktop, the live audio system that connects an untrusted voice model to an application through a virtual audio bridge, and the phone-call / voice-call-app skills built on top of live audio.

For the WebRTC capture/encode/stream stack that puts pixels on screen for a human viewer, see Display Pipeline. This page is about the model seeing and acting, not the human-facing video transport.

Computer Use

The CU abstraction (src/bin/caller/computer_use.rs) gives any provider a common action set and dispatches it through a platform backend. Provider- specific parsing of CU tool calls (OpenAI computer-use, Anthropic computer_20251124, Gemini) lives in the provider/ per-provider modules (openai.rs, anthropic.rs, gemini.rs); the executor here is provider-neutral. Anthropic wait/hold_key durations arrive in seconds and are converted to milliseconds (clamped to 30 s) at parse time.

The full CuAction vocabulary (the same tagged JSON accepted by the MCP execute_cu_actions tool and intendant ctl cu actions): click, double_click, triple_click, mouse_down, mouse_up, type, paste (clipboard + paste chord — fast for long text; previous clipboard text is restored where the platform allows, see “Action results” below), key, hold_key, scroll, move_mouse, drag, screenshot, zoom (region capture at the highest resolution the platform can supply — native 2x pixels on Retina), and wait. intendant ctl cu actions --help prints the per-action field shapes with an example.

Backends

DisplayBackend (in computer_use.rs) is detected at runtime by DisplayBackend::detect() — macOS → MacOS; Windows → Windows; otherwise WAYLAND_DISPLAY set → Wayland; else X11. It can be forced via the backend config value.

Wayland and Windows are session-only backends: both capture and input run through the live DisplaySession (WebRTC pipeline); without an active session the executor returns an actionable error naming the recovery path. X11 and macOS inject input directly — X11 uses a persistent x11rb/XTest connection with in-process root-window capture and clipboard support; macOS posts CoreGraphics CGEvents in-process (no cliclick/osascript subprocesses; key chords and typed ASCII use ANSI-US virtual keycodes, characters with no ANSI-US key ride paced unicode-string events, and typed text is read back via AX where possible — see “Action results” below) — and prefer the in-memory frame of a live capture session for screenshots, falling back to their platform capture paths when no session exists.

Post-action freshness: capture backends are damage-driven, so a screenshot taken right after an input action waits (bounded, 300 ms) for a frame captured after the action before falling back to the freshest available frame — the model never sees pre-click pixels after a click that changed the screen.

Coordinates from the model are in the provider’s logical screenshot space and are scaled to the backend’s actual pixel/point space before dispatch (important on HiDPI and under the Wayland portal, which reports its own stream size). zoom is the deliberate exception: on macOS it captures raw physical pixels (2x on Retina) and crops the requested logical region, because detail is the point.

Action results: dispatch vs. effect

OS input APIs only confirm that events were dispatched — not that the target app acted on them — so every action result carries an honest status (CuActionStatus) instead of a bare boolean:

• ok — the intended effect was verified: screenshots and zooms (the captured bytes are the effect), wait, and type on the macOS user session when the focused element’s AX value was read back and contained the typed text.
• injected — events were dispatched to the OS but the effect was not verified. This is the honest ceiling for clicks, keys, scrolls, drags, and typing on backends without read-back; verify from the post-action screenshot. A dispatched-but-ineffective action (a click that only activated an inactive window, a chord swallowed by the wrong app) reports injected, never ok.
• failed — dispatch failed, or verification contradicted the intent: a macOS type whose read-back does not contain the typed text returns failed with expected-vs-observed evidence rather than pretending success.

Type read-back is bounded and best-effort (two AX reads of the focused element): multi-line/control text (Return may submit, Tab may move focus), secure fields, and elements without an AX-readable value stay injected with the reason in the result detail.

Typing on macOS: ASCII is delivered as real ANSI-US keycode events (the same proven event shape as key), newlines as Return and tabs as Tab; characters with no ANSI-US key are delivered as paced unicode-string events (≤ 20 UTF-16 units per event, payload on both keyDown and keyUp, surrogate pairs never split). This replaces the 2026-07-13 failure mode where back-to-back CGEventKeyboardSetUnicodeString chunks were silently dropped by the focused app while the action still reported success.

Clipboard hygiene (paste): paste routes through the system clipboard, so each backend restores what it can and reports what it did in the result detail. macOS captures the previous text clipboard (pbpaste) and restores it ~300 ms after the chord — non-text content (images) cannot be captured, so the clipboard is cleared rather than left holding the paste payload; Windows does the same via arboard. X11 and Wayland selections are pull-based (the target may fetch the payload after the chord returns), so restore would race the paste itself: the pasted text deliberately remains the active selection and the result detail says so. The restore delay is an honest race, not a guarantee — an app that lazily re-reads the clipboard later sees the restored content.

Screen-capture permissions & signing identity (macOS)

macOS TCC keys every permission grant (Screen Recording, Accessibility, Apple Events, Microphone, Camera) to the app’s code-signing designated requirement as it was at grant time. If the binary is rebuilt or re-signed under a different identity — or the signing identifier changes — every existing grant silently stops validating: System Settings keeps showing the toggle ON while SCShareableContent::get fails with “The user declined TCCs…”. The macOS capture backend classifies that failure and appends the recovery path to the error (permission missing or invalidated by a re-signed build; toggle it off/on under System Settings → Privacy & Security → Screen & System Audio Recording, then relaunch — grants are only re-read at launch), and requests screen-capture access once per process so a fresh install gets the native prompt. The CU executor’s raw screencapture fallback gets the same treatment (cu_readiness::enrich_capture_failure): when the capture fails and CGPreflightScreenCaptureAccess confirms the permission is missing, the error names Screen Recording, the affected binary path, and the System Settings destination instead of the bare “could not create image from display”.

scripts/bundle-macos.sh defends the identity’s stability:

• The local-dev signing identity (“Intendant Dev”, in ~/.intendant/signing.keychain-db) is escrowed to ~/.intendant/signing-identity.p12 (chmod 600) when created, and backfilled from the keychain for identities that predate the escrow. If the keychain is ever lost, the script re-imports that file — same cert, same designated requirement, existing grants keep validating — instead of silently minting a fresh identity that voids them all.
• When a genuinely new identity must be minted (keychain and escrow both gone), the script prints a loud re-grant warning listing the System Settings steps.
• After every signing run it diffs the bundle’s designated requirement against ~/.intendant/last-signed-dr.txt and prints the same warning on any drift (this also catches signing-identifier changes and dev-cert ↔ Developer ID switches), then refreshes the cache.

Element Observation (read_screen)

read_screen is the cheap textual grounding path: a filtered accessibility tree of the frontmost application’s focused window — roles, labels, values, and logical-point frames, plus a one-line summary of other visible windows — typically a few hundred tokens versus ~1.5k for a screenshot. Clicks ground deterministically: click the center of a reported frame. Depth/node caps are always announced in the output, never silent.

Implemented for the user session on all supported platforms: macOS AX (src/bin/caller/ax.rs, raw accessibility-sys bindings — the documented Unix unsafe exception), Linux AT-SPI (src/bin/caller/atspi_read.rs), and Windows UIA (src/bin/caller/windows_uia.rs). It inspects the user’s session, not virtual display pixels, and requires the platform accessibility path to be available. Exposed as an MCP tool (in the core bootstrap profile) and as intendant ctl cu elements [--format json]. Pixels remain the fallback for visual verification and for apps with sparse accessibility trees (web views notably).

Long element values and window titles (a data: URL, a whole document in a text view) are length-capped once, centrally (computer_use::cap_screen_elements_texts, 80 chars) with a stable marker — prefix… [N chars total, #hash8] — so one long value cannot dominate the observation while staying distinguishable from other long values sharing its prefix. The platform readers deliberately do not pre-truncate; the full_values: true tool param (ctl cu elements --full-values) skips the cap for exact-value requests (Linux AT-SPI still bounds each text fetch at a documented 4096-char transport cap).

Who Can Call CU

Every agent Intendant runs or supervises reaches the same executor:

• Native agent — provider-native CU tool calls (cu_enabled), executed in the controller.
• Codex / Claude Code / Kimi Code (supervised) — the MCP bootstrap set (tool_profile=core) carries read_screen, take_screenshot, and execute_cu_actions; the injected $INTENDANT/INTENDANT_MCP_URL env makes "$INTENDANT" ctl cu ... work from their shells for the long tail. See External Agent Orchestration.
• Pi (supervised) — upstream Pi has no MCP client, so its appended supervision prompt uses the same scoped $INTENDANT ctl cu ... path directly. It reaches the same executor and grants; the UI never labels it as native Pi MCP.
• Anything with a shell — intendant ctl cu speaks to the running daemon’s /mcp endpoint; discovery is lazy via --help.

All paths execute under the same server-side autonomy/approval model (DisplayControl session grant for the user’s display).

Display Targets

CU actions operate on a DisplayTarget (#[serde(tag = "kind")]):

• Virtual { id } — an Xvfb-managed virtual display (:99, :100, …). display_env_string() → ":<id>". Virtual displays come into being three ways: the agent loop’s Xvfb auto-launch, the agent starting one itself (Xvfb :99 … &), or the dashboard’s keyless New virtual display action (create_virtual_display) — the path that gives an authorized headless box a display with no API key configured. Dashboard-created displays register a capture session immediately (streaming tile, CU-routable), are daemon-owned, and are destroyed when their tile is closed (a hard daemon kill leaves the usual orphan for the next allocation to reclaim). Xvfb is Linux-only; other platforms answer the action with a clear error.
• UserSession — the user’s real desktop. On Linux X11 it resolves the login session’s DISPLAY (falling back to :0); on macOS the primary display doesn’t use DISPLAY. Requires an explicit DisplayControl grant via the autonomy system — enforced fail-closed at the execute_actions chokepoint on every backend (the raw X11/macOS capture/injection paths included, not just by session existence on Wayland/Windows), with one exemption: an owner surface (an owner/root dashboard, an enrolled root user client, local loopback ctl, or the stdio MCP transport the owner wired up — ToolCallerTrust in mcp/mod.rs, derived from the bound AccessPrincipal) may target its own desktop ungranted, because the owner’s call is the opt-in. Every other caller — supervised external agents, scoped grants, federated peers — needs the standing grant.

When a pixel-CU call (take_screenshot, execute_cu_actions) omits display_target, the default is availability-aware (computer_use::default_display_target): the lowest-id live virtual capture session wins, then the conventional agent Xvfb display (:99) when its X socket is up (Linux only), then the user session. Explicit targets are never second-guessed, and every downstream gate applies to the fallback exactly as it would to an explicit user_session request. The native agent loop uses the same resolution when a session has no CU display configured, except its user-session fallback additionally requires the user-display grant — the model-driven path never auto-targets the user’s desktop ungranted. (read_screen defaults to the user session unconditionally — element trees only exist there — and sits behind the same grant/owner gate on all platforms: the element tree reveals window titles and field values just as pixels do, and it bypasses the session pipeline entirely.)

User-session access uses a session-grant model: approve once (the d hotkey the dashboard’s Share with agent action, MCP grant_user_display, or intendant ctl display grant-user), and the grant holds for the rest of the session until revoked. On Wayland, granting starts the GNOME portal flow. For Computer Use, the operator must enable Allow Remote Interaction in the physical portal dialog before clicking Share; approving screen sharing alone can produce screenshots while leaving keyboard/mouse injection unavailable. See Autonomy & Approvals for the approval surface.

CU Readiness Diagnosis (display_readiness)

The display grant is Intendant authority only — OS-level capability is a separate set of layers, and any of them can block actual CU while the grant reads as held (the classic macOS shape: already_granted, yet Screen Recording denies every capture). src/bin/caller/cu_readiness.rs probes five layers independently and names the non-ready ones, each with a fix:

1. intendant_display_authority — the user-display grant / caller trust (virtual targets need none).
2. screen_capture_permission — macOS Screen Recording (CGPreflightScreenCaptureAccess); Linux Wayland portal session or X11 socket; Windows desktop capture session.
3. accessibility_permission — macOS Accessibility (AXIsProcessTrusted); Linux AT-SPI bus reachability (bounded probe); Windows UIA client instantiation.
4. target_display — the requested display exists / has a live capture session.
5. input_backend — CGEvent (macOS, rides Accessibility), XTest (X11), portal remote desktop (Wayland), SendInput via session (Windows).

Probes are cheap, strictly read-only (they never pop permission prompts), and never cached — TCC and portal state can change or be revoked at any moment. A probe that cannot determine its layer reports unknown, and unknown counts as not ready (fail closed). Surfaces:

• the display_readiness MCP tool (display-view IAM class; listed in the core and screen profiles),
• intendant ctl display status [--target TARGET],
• request_user_display answers (already_granted and approvals) carry an os_readiness gap block naming any still-blocked OS layers, so a granted request can never masquerade as a CU-ready one.

Observation Policy (observe)

Every execute_cu_actions batch ends with a trailing observation, and what that observation is is a per-call policy (observe param; src/bin/caller/cu_observation.rs), not an unconditional screenshot:

• pixels (default) — the historical behavior: a post-action screenshot is captured and appended as one extra result. The default stays pixels because the tool description and the native peer cu guidance promise a screenshot; token-sensitive callers opt in to the cheaper modes (managed Codex’s developer instructions teach observe:"auto").
• ax — the frontmost element tree (the read_screen walk, same central text caps) is attached as text instead of pixels: a few hundred tokens versus ~1.5k image tokens, and no capture/encode work at all. User-session targets only (element trees exist nowhere else); a failed walk reports the error as the observation rather than silently substituting an image.
• auto — ax when the frontmost tree is usable (≥ cu_observation::AX_AUTO_MIN_NODES nodes), pixels otherwise (sparse tree, walk error, virtual-display target).
• none — per-action results only, for callers chaining batches that will observe once at the end.

The result always names the observation it carries and why (observation: pixels (auto: ax sparse (2 nodes) → pixels)), so a fallback is never silent. Two invariants regardless of mode: an explicit trailing screenshot/zoom action always returns its pixels (the action is the request), and the managed-context compact contract is preserved — path + metadata, no inline image bytes — with ax/auto the compact caller gets the element tree inline, which is the first time the managed path carries an actual observation instead of a path to fetch.

Screenshots are clean by default, encoded once. Click markers (crosshairs at click coordinates) are opt-in via annotate: true — baked-in markers obscured the very controls being verified (e2e finding CU-06), and the dashboard Live tab already overlays actions in real time. Annotation is drawn on the raw frame before the single PNG encode; the historical pipeline (capture → encode → decode → draw → re-encode → rewrite the disk artifact) is gone, and the disk artifact now always carries the same bytes as the model payload. That parity is deliberate: the artifact’s remaining reader is the managed-Codex view_image path (the Activity-tab disk-substitution consumer was retired with the Gemini CLI backend), which wants exactly what the model would have seen — including, on macOS, the logical-size (not raw-Retina) image that CU coordinates map to. Each batch logs a [cu] measurement line (observation kind/reason, capture+encode ms, AX walk ms, observation bytes, settle outcome) to the daemon log, and the native loop logs the same line into the session log.

Settle: bounded UI quiescence (settle)

The 300 ms freshness floor guarantees a recent frame, not a finished UI — after a click that starts a page load, the model historically padded batches with guessed wait actions. settle replaces the guess with a bounded quiescence wait (cu_observation): after the last input action, watch the display and return once no content change has been observed for a ~300 ms quiet window, capped at 2 s (settle: true) or a caller-supplied cap (settle: <ms>, clamped to 5 s; 0/false = off).

• Anchoring: the wait is anchored at the last input action. It runs before the batch’s first capture that follows an input ([click, screenshot] settles between the two), or before the trailing observation — the AX walk and the auto-screenshot both read the settled UI. A batch with no input actions settles from call start (“capture once quiet”).
• Damage signal: platform-native dirty rects when frames carry them (ScreenCaptureKit); otherwise a per-frame content fingerprint — the X11 backend polls at the capture rate and re-delivers unchanged frames, so frame arrival is deliberately not treated as change.
• Honest reporting: the result carries settle: settled after Nms (no display change for 300ms) or still_loading after Nms (display still changing at the cap). Paths without a usable damage signal — no live capture session, a capture stream that ends mid-wait, or the synthetic test-card backend (whose counter strip free-runs, which also keeps the e2e suite deterministic) — degrade to a fixed minimal wait and say so (fixed 300ms wait (...)).
• No double-wait: a damage-verified settle subsumes the capture freshness wait — the stream was watched past the input, so the following capture serves the current frame immediately.

settle composes with every observe mode (with none it simply bounds the batch’s return for callers chaining batches), and is exposed on the MCP tool, ctl cu actions --settle MS, and the peer twins (older peers ignore it).

Three separate concepts: private view, agent share, presence streaming

Putting the user’s screen on the wire means one of three deliberately distinct things, and the surfaces never conflate them:

1. Private view (dashboard View this machine) — remote view/control of this machine’s display from the owner’s dashboards. The capture session is created with agent_visible = false (ControlMsg::GrantUserDisplay { agent_visible: false }): it streams over WebRTC only to owner/root dashboard connections and accepts input only from those connections. The same ceiling is enforced on the legacy /ws path and the verified dashboard-control DataChannel; a scoped role’s display.view or display.input permission covers agent-visible displays only. The private view is invisible to agents, fail closed, at two independent layers:
   • the DisplayControl autonomy grant is never set (so the CU execute_actions chokepoint, INTENDANT_USER_DISPLAY_GRANTED on runtime children, shared_view, and read_screen all refuse the user session exactly as if nothing was granted), and
   • the session itself is skipped by every agent-facing registry lookup — SessionRegistry::get / display_ids() are filtered by the session’s agent_visible flag, so CU session routing, the screenshot session lookup, default_display_target, list_displays resolution overlays, federated peer streaming and peer display announcements, presence’s available-display list, and the 1 Hz FrameRegistry sampler (display_<id> model-feed frames, hence list_frames/read_frame and conversation auto-attach) all read it as absent. Owner/root dashboard paths may use the unfiltered get_any / all_display_ids accessors after that authority check; scoped dashboards and federated peers stay on the filtered accessors. Auto-recording also skips private views. For the alpha, an explicit StartRecording is owner/root-only and still requires an active agent-visible display. A private or absent display is rejected before ffmpeg starts: recording artifacts live in the ordinary session tree, so private pixels must never be written there in the first place.
2. Agent share (dashboard Share with agent, MCP grant_user_display, ctl display grant-user) — the classic grant: agent_visible = true, the autonomy grant is set, and the display is enumerable/screenshotable/drivable by agents until revoked. A share over an existing private view upgrades it in place (frames start feeding the registry on the next sampler tick; no capture restart, no second portal dialog). The reverse never happens implicitly: a view request over a shared display does not downgrade it — taking the agent’s access away is an explicit revoke.
3. Presence streaming (the tile’s Stream button) — continuous frames to the live presence (voice) model only. Independent of both modes above and unchanged by them; main agents are not affected.

Revoking a user display (either mode) tears the session down and clears the per-daemon grant flag. On the wire, GrantUserDisplay without agent_visible keeps its historical meaning (true — the agent share), so pre-split frontends and scripts are unaffected.

Granting and resolving a display request are themselves owner/root-only: GrantUserDisplay and ResolveDisplayRequest from a scoped caller are refused on both dashboard transports. RevokeUserDisplay stays open to an otherwise authorized scoped caller — de-escalation is fail-safe. The shared-view tools (show_shared_view, focus_shared_view, request_shared_view_input, capture_shared_view_frame) activate a user-session target only for a granted or owner caller instead of flipping the grant themselves. Sharing an agent-owned virtual display never touches the user-display grant.

Dashboard Live workspace

The dashboard’s Live tab presents local Computer Use as one selected display stage. Every active DisplaySlot and its WebRTC connection remain alive, but only the selected slot is visible; the rail switches that projection without recreating the capture or video element. Agent-requested shared views select their target when the current stage has no active human work. They remain advisory when the user is controlling, awaiting input, annotating, calling out, or viewing full-screen; the banner and rail keep the requested target discoverable without discarding that work. Peer-display launchers are different: they route to the peer-aware Station surface before opening the stream rather than pretending a federated pane is local.

Input authority in the stage and rail is always the browser-relative server state: you, another viewer, available (unclaimed), or connecting. Take is last-take-wins; no holder identity or approval ceremony is inferred. Only one hidden-input hazard is handled locally: selecting another display or leaving Live releases an actively bound slot after flushing held keys and mouse buttons, then removes its keyboard, pointer, and document-level paste listeners. Annotation and armed-callout state is editable work, so display and tab navigation is blocked with a visible explanation until the user finishes or closes it. Pending Take requests and already-held-but-locally-unbound authority are also cancelled before a surface can be hidden. Activity’s shared-view Take input returns to the full Live stage before requesting authority; thumbnails remain view-only.

The rail’s per-display activity feed combines browser-observed lifecycle events (stream connection, authority, private/share mode, presence streaming, recording, annotation, and shared-view focus) with the daemon’s live action lane: computer_use::execute_actions emits one cu_action event per successfully executed CU action (CuActionObserver), carrying the kind (left_click, type, screenshot, …), display id, driving session id, display-space coordinates plus their reference resolution, a short raw call string (left_click(612, 233); embedded text truncated for presentation — the Activity log keeps the full trace), a unix-ms timestamp, and a dedupe event_id. The lane is deliberately ephemeral: it rides the outbound broadcast to the /ws and dashboard-control lanes only — never the session log, never replay — and mouse moves coalesce to 10 Hz. Failed actions never emit; the overlays must not show clicks that did not happen. Peer upcasters drop the event, so federated (peer) displays render no action overlays today — a known follow-up.

Those events drive the stage’s action overlays on the display they belong to: an agent cursor (white arrow + verb pill reading Look / Move / Click / Type / Scroll / Waiting) that eases toward each action point, dims while the dashboard user holds input authority, and fades out when idle; a click ripple; last-typed keypress chips (from the truncated raw call, space shown as ·); and a full-stage screenshot flash. The concept’s target-highlight box + role tag are deliberately not implemented — no element/role data exists on the wire (future AX integration). All overlays are pointer-events: none, aria-hidden, honor prefers-reduced-motion, and compute geometry from the letterboxed video rect — nothing touches the video element itself. Feed rows for actions use a two-line grammar (friendly sentence + seconds-precision timestamp, raw mono call below); the feed keeps the last 50 entries per display and follows the bottom only when already scrolled there.

cu_action session attribution also feeds the rail’s approval card: when a pending approval belongs to the session the daemon last reported driving the selected display, an amber card renders under Input authority whose Approve/Deny proxy the main approval panel’s own session-scoped actions. No attribution, no card — it never guesses. Recording transitions enter the feed only after the daemon confirms them; Start, Stop, and Delete remain pending and retain the last confirmed state on transport error or timeout, and while recording the Record button ticks the elapsed time from the confirmation. At narrow widths the same rail becomes a keyboard-accessible modal drawer; the stage toolbar remains the fallback for primary controls. In-page full screen similarly contains focus, hides its background from assistive technology, supports Escape, and restores the invoking control.

CU-First Routing

Status note (vaulted 2026-07-04): the all-tasks CU-first interception — where every non-direct task in the agent loop is first offered to a fast CU model (try_cu_first in display_glue.rs) — is off by default, behind [experimental] cu_first_routing = true. In practice it added a model hop (latency) to every task and, under subscription-based external agents (Codex, Claude Code, Kimi Code, Pi), reintroduced an API-key model the deployment otherwise didn’t need. The code stays runnable for a future pickup. What remains always-on is the frame-grounded dispatch described below: when the user issues a task while pointing at a display, that is an explicit computer-use request, and the CU task path is the only machinery that can act on the referenced frames. The fast paths and observation layer above are designed for the heavy agents (native, Codex, Claude Code, Kimi Code, Pi) as primary consumers and do not depend on either router.

Frame-grounded work goes to a fast CU model, with escalation to the heavy agent for anything that turns out to need code changes. The routing decision lives in the session supervisor (session_supervisor/launch.rs, start_new_session → spawn_cu_task):

submit_task / StartTask
        │
        ▼
 reference_frame_ids non-empty?
        │
        ├── yes ─▶ spawn_cu_task  (fast CU model, with the referenced frames
        │             │            resolved to images as visual context)
        │             │
        │             └── CU model decides it's not a display task
        │                  → calls escalate_to_agent → heavy agent runs it
        │
        └── no  ─▶ normal agent / orchestrator path

The gate is reference_frame_ids being non-empty (the frames the user was looking at, supplied by presence’s submit_task); the display_target hint is carried through to tell the CU pipeline which display to act on. Earlier docs implied display_target alone triggers CU routing — the actual trigger is the presence of reference frames. The CU provider is given native CU tools plus a single escalate_to_agent function tool; calling it ends the CU run with CuTaskResult::Escalate and hands the task to the main agent.

Configuration

[computer_use]
provider = "gemini"          # optional; default = CU_PROVIDER / PROVIDER env, else auto
model = "gemini-3-flash-preview"  # optional; gemini default shown
backend = "auto"             # "x11" | "wayland" | "macos" | "windows" | "auto" (default)

Provider/model resolution (provider::select_cu_provider): config → CU_PROVIDER / CU_MODEL env → PROVIDER env → auto. Default models when unset: gemini gemini-3-flash-preview, anthropic / openai use their CU-capable defaults.

Live Audio

spawn_live_audio is an agent tool (defined in src/bin/caller/tools.rs), not a CLI flag. It spins up an untrusted voice sub-agent that talks to Gemini Live or OpenAI Realtime and exchanges audio with an application through a virtual audio bridge.

How It Works

voice model ──WebSocket──▶ Intendant ──audio bridge──▶ application
   │  PCM16 mono 24 kHz        │   virtual mic/speaker      │
   │  structured tool calls    │   (platform audio bridge) │
   │◀─────────────────────────│◀──────────────────────────│

1. The agent calls spawn_live_audio with id, provider, playbook, and a mandatory response_schema (plus optional timeout_secs, voice, display_id, initial_message).
2. Intendant opens an audio bridge and connects to the voice model with a whitelisted tool set generated from the response schema.
3. The model follows the playbook; its turns are bridged as audio to/from the app, and inbound audio is also teed to Whisper for a transcript.
4. When the model calls submit_response, the data is validated against the schema; the tool returns a LiveAudioResult with a status of Completed, TimedOut, Disconnected, SchemaError, or Failed (the last two carry an error string).

The live path is bounded for real-time behavior. Vortex capture still polls its shared-memory ring every 5 ms, but aggregates samples into 20–40 ms WebSocket frames and flushes a short tail when the input goes quiet. Playback, capture, and the optional Whisper tee each hold at most 30 s of PCM16 and evict the oldest audio on overflow, so a stalled consumer skips ahead instead of growing memory or replaying a call far behind real time. Whisper intake forms 3 s windows, keeps at most four pending while one sequential request is in flight, drops the oldest completed window on overflow, and flushes each ordered transcript entry to disk.

Security Model — Untrusted, Schema-Validated, Quarantined

The voice model is treated as hostile input. It has zero tools beyond the two generated from the schema (submit_response, end_call) and zero file access. Three layers protect the rest of the system:

• Whitelisted tools + schema validation (schema_validator.rs): the model can only call the response tool; submitted data is checked against the declared ResponseSchema, with oversized fields truncated and off-schema data rejected.
• Quarantine (quarantine.rs): any unexpected content — a tool-call attempt for an unknown tool, oversized strings, off-schema payloads — is written to ~/.intendant/quarantine/<live_audio_id>/<payload_id>.json and only a reference is returned; the raw content is never surfaced to the agent.
• Sandbox: when the write sandbox is enabled, live-audio processes can write only to the session log and quarantine directories — no project root, no /tmp.

Silence Watchdog

The session loop runs a time-based watchdog (live_audio.rs): if there has been no model output for 15 seconds, it sends one nudge (“Are you still there? Please continue the conversation.”) to unstick a frozen model, and resets when output resumes. A separate turn counter nudges the model toward emitting its JSON response after enough turns. (Earlier docs described “6 consecutive unresponsive turns” — the real mechanism is the 15-second silence timer.)

Audio Bridge — Per Platform

On macOS and Linux the bridge first probes for the Vortex shared-memory segment (shm_open("/vortex-audio")). The preferred Unix path is the Vortex Audio HAL plugin via a direct POSIX shared-memory bridge (start_vortex_shm_bridge, shm_open + mmap, no daemon/socket); if its segment is absent, the platform fallback is used. Windows goes directly to its ffmpeg/VB-CABLE fallback:

Audio routing is only needed for the app-to-model bridge. Browser voice interaction through the dashboard (Gemini Live / OpenAI Realtime) needs none of this.

Configuration

[live_audio]
enabled = false                # default: false
default_timeout_secs = 300     # default: 300 (5 minutes)
gemini_model = "gemini-2.5-flash-native-audio-preview-12-2025"  # optional
openai_model = "gpt-4o-realtime-preview"                        # optional
sample_rate = 24000            # default: 24000

LiveAudioSpawn is its own autonomy category, so spawning a voice session can be gated independently of other actions.

Phone Calls (phone-call skill)

skills/phone-call/SKILL.md places an outbound SIP call and conducts the conversation with a voice model. macOS only; requires the Vortex Audio HAL plugin, pjsua, and a GUI session with TCC mic permission.

voice model ──shm──▶ Vortex Audio device ──▶ pjsua (SIP/SRTP) ──▶ phone
   │                                                                          │
   │◀────────────────────────────────────────────────────────────────────────│

How it works:

1. Find the Vortex device index (pjsua --null-audio | grep vortex).
2. Start pjsua with Vortex as both --capture-dev and --playback-dev, SRTP enabled, dialing the target SIP URI.
3. Immediately call spawn_live_audio (provider: openai) — do not wait for the call to connect; the shm bridge polls and works before connect.
4. The model conducts the call per the playbook and returns structured data.
5. Clean up pjsua.

response_schema is mandatory — without it the call is rejected with a parse error. Do not set initial_message: the model starts speaking when it hears the callee.

Voice Calls Through Any App (voice-call-app skill)

skills/voice-call-app/SKILL.md makes a voice call through any app (Element, FaceTime, WhatsApp, …) by combining computer use to drive the UI with spawn_live_audio for the conversation. macOS or Linux with a display; requires the Vortex Audio HAL plugin and a GUI/TCC mic permission.

How it works:

1. Prepare the spawn_live_audio arguments (playbook, schema, voice, id) before dialing, so they fire the instant the call connects.
2. Use CU actions to foreground the app, navigate to the contact, and click the call button. (take_display_control is not required for execute_cu_actions — only take it if you need exclusive input.)
3. Call spawn_live_audio (ideally in the same turn as the call click to minimize dead air).
4. Write the result from response_data immediately; hang up on completion.

The voice model has two generated functions here: submit_response (the schema fields) and end_call. It submits data, then signals end_call.

Response Schema Format

Both skills use the same ResponseSchema shape (live_audio_types.rs). Each field nests its type under field_type:

{
  "fields": [
    {"name": "guest_name",       "field_type": {"type": "string",  "max_length": 100, "tainted": true}, "required": true,  "description": "Guest name"},
    {"name": "party_size",       "field_type": {"type": "integer", "min": 1, "max": 50},                "required": true,  "description": "Number of guests"},
    {"name": "reservation_time", "field_type": {"type": "string",  "max_length": 50,  "tainted": true}, "required": true,  "description": "Confirmed time"},
    {"name": "confirmed",        "field_type": {"type": "boolean"},                                     "required": true,  "description": "Whether confirmed"},
    {"name": "special_requests", "field_type": {"type": "string",  "max_length": 200, "tainted": true}, "required": false, "description": "Any special requests"}
  ]
}

Field types: string (max_length, allowed_values, tainted), integer (min, max), boolean, array. The voice model cannot submit until all required: true fields are filled. Fields marked tainted: true carry user-/callee-provided content and are treated as untrusted data, never as instructions.

Browser Microphone Transcription

Separately from live audio, the server can transcribe the user’s dashboard microphone via Whisper (transcription.rs). Off by default. The web gateway buffers user_audio WebSocket frames into ~3-second chunks, filters silence by RMS energy, wraps them as WAV, sends them to the transcription API, and broadcasts user_transcript events.

[transcription]
enabled = true
provider = "openai"
model = "whisper-1"          # default
language = "en"              # optional ISO-639-1 hint
# endpoint = "http://..."    # optional; custom/self-hosted whisper-compatible endpoint

Requires OPENAI_API_KEY (or a custom endpoint).