Integrations

Intendant integrates with the outside world in several directions: it consumes external MCP servers and external coding-agent CLIs as tools, routes audio through platform virtual-audio bridges, transcribes speech, records displays, and shells out to a set of system tools. This chapter covers each, plus the control socket and web gateway programmatic surfaces and the CI / setup tooling.

For the MCP server (exposing Intendant’s own control surface as MCP tools) see MCP Server. For the presence layer see Presence Layer.

MCP client

Intendant can connect to external Model Context Protocol servers and surface their tools to the agent. Servers are declared as mcp_servers entries in intendant.toml:

[[mcp_servers]]
name = "filesystem"
command = "npx"
args = ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]

[[mcp_servers]]
name = "github"
command = "npx"
args = ["-y", "@modelcontextprotocol/server-github"]

[mcp_servers.env]
GITHUB_TOKEN = "ghp_..."

Each entry has name, command, optional args, and an optional env table (see Configuration). At startup each server is spawned as a child process, its tool list is fetched, and every tool is registered to the agent under the name mcp__<server>_<tool> (e.g. mcp__github_list_issues). When the agent calls one of these, the client routes the call to the right server by parsing that prefix.

Every outbound mcp__* call passes the controller’s tool_call approval gate before dispatch. The shipped tool_call = "ask" rule therefore prompts at Medium autonomy; owners who deliberately trust their configured servers can set it to auto.

Replies cross a separate context boundary. Intendant preserves the MCP is_error status and wraps text (including transport error text) as explicitly untrusted, quoted data before adding it to the model conversation. Control, invisible-format, and bidirectional-override characters are made visible; carriage returns are normalized; and the complete rendered result is capped at 64 KiB with an explicit truncation marker. Non-text and structured payloads are not copied into this text bridge. These measures bound and label the input; they do not make an external server’s claims trustworthy.

Trust model

This is a privileged integration with no sandboxing:

An mcp_servers entry is spawned with your full privileges — Command::new(&config.command).args(&config.args) in mcp_client.rs. Intendant performs no checksum verification, no signature check, and no sandboxing of the server binary. Adding one is equivalent to adding a line to your ~/.zshrc that runs a binary.

The default is mcp_servers = [], and intendant.toml is git-ignored, so the repository ships no MCP servers. Treat copying an intendant.toml between machines like copying shell rc files: read it before you source it. The same trust framing applies to the MCP server side — see MCP Server.

External coding-agent CLIs

Instead of the native agent loop, Intendant can delegate coding work to an external harness — Codex, Claude Code, Kimi Code, or Pi — selected per-invocation with --agent <codex|claude-code|kimi|pi> or by default via [agent] default_backend. Each backend’s binary path, model, sandbox/approval policy, and tool restrictions are configured under [agent.codex], [agent.claude_code], [agent.kimi], and [agent.pi] (full key reference in Configuration).

Notable details:

• Codex runs against its app-server (thread/start); the sandbox, approval_policy, reasoning_effort, web_search, network_access, and writable_roots keys map onto Codex’s own CLI flags. Approval prompts from Codex are surfaced through Intendant’s normal approval UI.
• Claude Code runs its stream-json protocol over stdio. Intendant lifts tool approvals and structured questions, preserves its native session id, and surfaces async Agent/Task children as session relationships.
• Kimi Code runs an isolated local kimi server and uses its authenticated REST/WebSocket surface. This retains native queued steering, undo, fork, :btw/swarm sub-agents, goals, live model/thinking/permission/plan/swarm changes, structured interactions, file upload, and reconnect snapshots that Kimi’s narrower ACP facade does not expose.
• Pi runs upstream’s documented JSONL RPC mode with discovered extensions disabled and one private fail-closed Intendant approval extension. It keeps Pi’s native subscription/API provider support, sessions, images, steering, interruption, compaction, fork, rename, and live model/thinking controls. Pi has no built-in MCP; the supervised prompt truthfully directs it to the scoped $INTENDANT ctl bootstrap for platform capabilities.

This is a deep topic with its own chapter — see External Agent Orchestration.

Audio stack

Live voice and phone calls pipe audio through a virtual audio bridge so the voice model’s I/O can be captured and injected independently of the host’s real devices. The backend is chosen per platform (audio_routing.rs):

                live audio / phone-call session
                              │
       ┌──────────────┬──────────────┬──────────────┬─────────────────┐
  Unix + Vortex   Linux fallback  macOS fallback   Windows fallback
       │               │               │                 │
  Vortex HAL shm  PulseAudio       BlackHole       ffmpeg DirectShow
  rings            null sinks       + sox           + ffplay/VB-CABLE

• Unix preferred: Vortex Audio. The Vortex HAL plugin shipped in vendor/vortex-guest-tools/ exposes a “Vortex Audio” device to apps. Intendant probes /vortex-audio and exchanges PCM through the plugin’s POSIX shared-memory rings directly — no daemon, no socket, no system audio reroute. scripts/setup-macos.sh installs it on macOS (or scripts/update-vortex-pkg.sh updates the package); see Computer Use & Live Audio.
• Linux: PulseAudio null sinks. pactl creates null sinks for the mic and speaker paths, and the session temporarily swaps the default source/sink, restoring them on teardown. (pulseaudio-utils provides pactl.)
• macOS fallback: BlackHole. When Vortex is not present, BlackHole 2ch / 16ch virtual devices are driven with SwitchAudioSource and sox.
• Windows fallback: ffmpeg + VB-CABLE. ffmpeg captures the cable’s CABLE Output recording endpoint through DirectShow; ffplay plays model PCM through the system default output. The user must install VB-CABLE and route the default output to CABLE Input; there is no automatic per-app route or default-device save/restore.

The phone-call skill places outbound SIP calls via pjsua, with the live voice model conducting the conversation and returning structured data. The live audio model is untrusted: zero tools, zero file access, and its outputs are schema-validated and quarantined.

Transcription (Whisper)

Server-side speech-to-text uses the OpenAI Whisper API (or a compatible endpoint) when [transcription] is enabled or --transcription is passed. The browser streams PCM16 audio; the gateway buffers ~3s chunks (buffer_secs, RMS-filtered to drop silence), wraps each in WAV, and POSTs it to the endpoint. Transcripts are broadcast as user_transcript events and logged. Configure via [transcription] (provider, model, language, endpoint) — see Configuration and Web Dashboard. Point endpoint at a self-hosted whisper.cpp server for a fully local pipeline.

Recording (ffmpeg)

Display recording is driven by ffmpeg and configured under [recording] (enabled, framerate, segment_duration_secs, quality, max_retention_hours). Recordings are segmented and served by the dashboard’s /recordings/* and /api/session/{id}/recordings/* endpoints, with timeline seeking and 1x/2x/4x playback in the Video tab. If [recording] enabled = true but ffmpeg is not installed, recording is disabled with a logged warning rather than failing the run. The --record-display <ID> flag records an existing X11 display (:ID) and is repeatable.

System tools

Intendant uses platform APIs and tools for capture, input injection, and search. Install them via the setup scripts (next section); the agent degrades gracefully with a clear error when one is missing.

X11 displays are auto-launched via Xvfb when the agent first needs one. See Display Pipeline for the capture/encode pipeline.

Setup and CI tooling

Setup scripts (scripts/)

intendant setup browsers can also be run directly to install or repair the managed browser cache used by CDP browser workspaces.

A legacy scripts/intendant-ctl.sh helper exists for the raw Unix control socket. It selects the first /tmp/intendant-*.sock, so it is not multi-instance safe and should be limited to one-instance protocol debugging. The supported command-line client is the binary itself: intendant ctl --help discovers the HTTP MCP control surface, and each command group has focused help.

When you add a new -sys crate dependency, update both scripts/setup-linux.sh (APT_PACKAGES) and scripts/setup-macos.sh in the same commit — otherwise fresh-machine setups break later with cryptic pkg-config errors.

CI (.github/workflows/)

main on github.com/intendant-dev/Intendant is ruleset-protected: every landing is a pull request validated by GitHub’s merge queue, and the checks marked required below gate it. The required-check workflows run unfiltered on pull_request and merge_group on purpose: GitHub only adds a PR to the queue once the PR’s own required checks pass, so a paths:-skipped required check blocks queue entry (and a skipped group check wedges the entry at “Expected”). Cheap integrity/docs workflows retain paths-filtered push triggers; the heavy cross-platform workflow has no push trigger because the merge group just validated the same tree.

Trusted refs (pushes, merge-queue refs, same-repo PRs) run on a self-hosted fleet (dell-206 / intendant-linux, macbook-a / intendant-macos, samsung-win / intendant-windows). Its incremental build state lives in external, per-listener CARGO_TARGET_DIR caches keyed by the Rust version; checkout cleanup wipes any in-workspace target/. Fork PRs route to GitHub-hosted runners via a dynamic runs-on (the matrix os key doubles as the hosted label): external code never executes on the fleet, while its required checks still genuinely run — a skipped required check would read as satisfied, so routing beats skipping. Fork-PR workflows additionally require maintainer approval before running at all (the Actions fork-approval policy covers all outside collaborators, not just first-timers), and only maintainers can enqueue. The Dell and Windows runners run as dedicated non-admin ci users (sudo-less on Linux; a plain Users-group service account on Windows), runners are registered per-repo, and the default GITHUB_TOKEN is read-only.

tests/e2e/main.rs contains hermetic #[tokio::test] cases. They spawn the real binaries against the scripted mock provider, use the synthetic 1280×720 display rather than native capture, need no API key or network, and run in CI on all three supported operating systems. The tests/skills/ scenarios that make real API calls or need a native display (the live claude-code-e2e battery, browser/Station probes, voice/audio scenarios) stay outside CI and run on operator hardware. Run the full keyless battery from Getting Started before committing: the three package binaries, workspace library gates, scripted-provider E2E suite, and cargo clippy --workspace -- -D warnings. The TLS stack is pure-Rust ring / rustls / rcgen (no OpenSSL), which is why the Windows CI job installs NASM (for ring’s assembly) but no libssl.

CLI discovery descriptor

A gateway-serving daemon records at boot where its controller binary lives, so unsupervised local agents (cron jobs, plain shells, agents outside Intendant supervision) can resolve a CLI even when intendant is not on PATH (app installs name the bundled controller intendant-bin and install no PATH shim). Under the state root ($INTENDANT_HOME or ~/.intendant):

• cli-path — one line, the absolute controller path; readable from a shell one-liner, no jq;
• cli-path.meta.json — debug context only (controller, port, pid, version, write time). Never secrets.

Written atomically on daemon boot only — ctl and transient invocations never touch it; on multi-daemon homes the last-booted daemon wins, which is fine because any controller binary can ctl any daemon at the standard endpoints (the sidecar identifies the writer for debugging). Every repo skill that shells to the CLI resolves it through the canonical preamble — $INTENDANT → PATH → this descriptor → bare intendant — and a repo test pins that preamble so future skills cannot regress.

Control socket

When --control-socket is enabled, a Unix domain socket is created at /tmp/intendant-<pid>.sock for programmatic control of a running instance from external scripts and tools. It is opt-in.

• Outbound: events are broadcast (newline-delimited JSON) to all connected clients.
• Inbound: newline-delimited JSON commands for status, approval/denial, human input, autonomy change, quit, controller-restart workflow, and (in MCP mode) controller-loop intervention.

Representative inbound commands (JSON-line)

{"action": "status"}
{"action": "approve", "id": 123}
{"action": "deny", "id": 123}
{"action": "input", "text": "answer to askHuman"}
{"action": "set_autonomy", "level": "high"}
{"action": "schedule_controller_restart", "controller_id":"codex", "north_star_goal":"audit and improve", "restart_after":"turn_end"}
{"action": "controller_turn_complete", "restart_id":"<id>", "turn_complete_token":"<token>", "status":"ok", "handoff_summary":"..."}
{"action": "get_restart_status"}
{"action": "cancel_controller_restart", "restart_id":"<id>"}
{"action": "request_controller_loop_halt", "persistent": true}
{"action": "clear_controller_loop_halt"}
{"action": "intervene_controller_loop", "mode":"stop"}
{"action": "get_controller_loop_status"}
{"action": "query_detail", "scope": "diff"}
{"action": "query_detail", "scope": "file", "target": "src/main.rs"}
{"action": "search_transcripts", "keywords": ["auth", "login"]}
{"action": "usage"}
{"action": "quit"}

Outbound events

{"event": "turn_started", "turn": 5, "budget_pct": 12.3}
{"event": "agent_output", "stdout": "...", "stderr": "..."}
{"event": "approval_required", "id": 123, "command": "rm -rf /tmp/test"}
{"event": "ask_human", "question": "Which database?"}
{"event": "task_complete", "reason": "done signal"}
{"event": "status", "turn": 3, "phase": "thinking", "autonomy": "medium", "session_id": "abc-123", "task": "fix tests"}
{"event": "usage", "main": {"provider": "openai", "model": "gpt-5.5", "tokens_used": 12000, "context_window": 128000, "usage_pct": 9.4}}
{"event": "usage_update", "main": {"provider": "openai", "model": "gpt-5.5", "tokens_used": 15000, "context_window": 128000, "usage_pct": 11.7}}
{"event": "display_ready", "display_id": 99, "width": 1024, "height": 768}
{"event": "command_result", "action": "get_restart_status", "ok": true, "message": "ok", "data": {}}

• status includes session_id and task. The usage event answers {"action":"usage"}; usage_update is broadcast automatically after each turn (with a presence field when the presence layer is active). display_ready fires when a display becomes available for WebRTC streaming; its display_id is numeric. Request parameters that target a display use strings such as display_99 in their own fields. command_result.ok is false when a control action fails (e.g. schedule_controller_restart with restart_after="now" and no executable restart action).

Examples

# Raw, opt-in Unix control-socket protocol:
echo '{"action":"status"}' | socat - UNIX:/tmp/intendant-$(pgrep intendant).sock

# Supported CLI (HTTP MCP, default port 8765; add --port PORT when needed):
intendant ctl status
intendant ctl approval pending
intendant ctl approval approve 123
intendant ctl input respond "answer to the pending question"
intendant ctl task start --task "new task description"

Web gateway

The default-on web gateway (--web, see Web Dashboard) serves the SPA and bridges WebSocket connections to the EventBus. It is the same control surface as the control socket, plus terminal I/O, presence, and WebRTC signaling.

Browser ──WebSocket──> Intendant web gateway (default port 8765)
  │                              │
  │  Tool requests               │  Events (broadcast to all clients)
  │  presence_connect/disconnect │  Tool responses (per-connection direct channel)
  │  Voice logs/checkpoints      │  State snapshot + log replay (on connect)
  │  Audio for transcription     │  Presence welcome (on voice connect)
  │  WebRTC signaling            │  WebRTC signaling (SDP, ICE)
  v                              v
App dashboard (WASM)        EventBus + AgentStateSnapshot
  +                              │
Optional: browser-side           │  Dual outbound channels:
live model (Gemini/OpenAI)       │  - broadcast::Receiver (events)
  │                              │  - mpsc::unbounded (direct responses)
  │  (function calls → tool_request)
  v
Intendant agent loop

The gateway has two layers:

1. App dashboard — the SPA at /, state managed by presence-web WASM. Events are broadcast; late-connecting browsers get a full log replay.
2. Presence bridge (optional) — a browser-side live model’s tool calls become tool_request WebSocket messages handled server-side, with tool_response returned on the per-connection direct channel.

WebSocket protocol

The tables below show the principal messages, not every dashboard-control, annotation, clip, and transport-maintenance variant.

Representative inbound (browser → server)

Representative outbound (server → browser)

Tool request/response and bootstrap

A browser live model calls presence tools via tagged request/response:

// Browser:
{"t":"tool_request","id":"req-42","tool":"check_status","args":{}}
// Server (direct channel):
{"t":"tool_response","id":"req-42","result":"Phase: Running agent (turn 5). Budget: 23% used."}

• Action tools (submit_task, approve_action, deny_action, skip_action, respond_to_question, set_autonomy, send_message) dispatch through the EventBus — the same path as control-socket commands.
• Query tools (check_status, query_detail, search_transcripts) are handled asynchronously server-side, reading the shared AgentStateSnapshot, project-confined files, and the current session’s voice transcript/event-log history.
• Video tools (inspect_frame, inspect_frames) examine frames from the frame registry.

On connect the server queues a deterministic bootstrap before opening the live broadcast lane: state_snapshot; cached usage/status and per-session state; cached autonomy, backend, and user-display state; pending display requests; the browser_workspace_snapshot; live display_ready frames; session/external-log replay; and, last, the current per-display input-authority snapshots. Optional cached entries are omitted when absent. This ordering recreates display slots before replay and then lets the final authority snapshot win over historical transitions.

HTTP endpoints

See Web Dashboard → HTTP endpoints for the endpoint table.

Requirements

• Microphone access requires a secure context — localhost (e.g. ssh -L 8765:localhost:8765 host), default HTTPS/mTLS, --tls with a trusted certificate, or the macOS app’s intendant:// scheme. See Web Dashboard → Secure browser contexts and LAN access.
• Voice credential (optional) — Gemini’s long-lived key stays in the unlocked client vault or legacy per-origin localStorage and goes directly from browser to Gemini. For OpenAI, the daemon holds or leases the long-lived key and mints the browser a short-lived Realtime client secret through api_voice_session / POST /session.

Supported tools (browser live model)