Display Pipeline

Intendant gives agents graphical displays they can see and interact with. The display pipeline streams a display’s frames to one or more browsers over a custom WebRTC transport with low latency, and routes remote keyboard/mouse input back the other way. A parallel tile-streaming path replaces whole-frame video with dirty-region updates on capable platforms.

This chapter covers the post-redesign architecture: one capture backend per display fanning out to a shared multi-codec encoder pool, with each browser peer driven by its own sans-I/O rtc peer connection. For sharing a remote peer’s display across machines, see Peer Federation. For the Windows-specific backends, see Windows Support.

Overview

A DisplaySession (crates/intendant-display/src/lib.rs) owns one display’s whole pipeline: the platform capture backend, a broadcast fan-out of raw frames, the encoder pool, the set of connected WebRTC peers, the clipboard monitor, and the tile-streaming bridge. The high-level data flow:

                    ┌──────────────────────────────────────────────┐
                    │              DisplaySession                    │
                    │                                                │
[CaptureBackend] ─mpsc(4)─▶ capture bridge ─broadcast(16)─▶ pool-feed bridge
  (X11/Wayland/             │  (Arc<Frame>)                   (BGRA→I420)
   macOS/Windows)           ▼                                      │
                       latest_frame                                ▼
                       (RwLock)                          ┌──────────────────┐
                            │                            │   EncoderPool    │
                            │                            │  baseline + sim- │
                            ▼                            │  ulcast + on-    │
                    tile-stream bridge                   │  demand encoders │
                    (XDamage / frame-diff)               └────────┬─────────┘
                            │                                      │ broadcast(16)
                            │ data channels                       │ Arc<EncodedFrame>
                            ▼                                      ▼
                    ┌───────────────────────────────────────────────────────┐
                    │  per-peer WebRtcPeer driver task (one per browser)      │
                    │  rtc PeerConnection + sockets · picks codec/layer ·     │
                    │  packetizes RTP · pumps UDP/TCP · TWCC tap              │
                    └───────────────────────────────────────────────────────┘
                            │ encrypted media + input data channels
                            ▼
                          browser

Two kinds of displays go through the same lifecycle:

• Virtual displays — on Linux, Xvfb displays (:99, :100, …) launched lazily when the agent first runs a graphical command. There is no Xvfb analogue on macOS or Windows.
• User-session displays — the user’s real desktop (:0 on Linux, the native display on macOS/Windows), opt-in via the DisplayControl autonomy category.

Backpressure rules

Every stage is bounded and lossy by design — slow consumers drop frames rather than back-pressuring the capture backend (which would degrade every other viewer):

The Encoder Pool

The redesign’s centerpiece (crates/intendant-display/src/encode/pool/: vocabulary types, the subscription lease + keyframe coalescer, the orchestrating EncoderPool, and the encoder worker threads). The pre-pool design used one encoder per session with the codec locked to the first peer’s offer: every later viewer had to accept that codec or its WebRTC offer failed outright. The naive fix — one encoder per peer — is what transcoding gateways do, and it scales badly (N× CPU; hardware encoders hit their session limit at ~5-8 viewers and silently fall back to software). The pool instead follows the production SFU pattern: a small shared bank of encoders that all peers consume, with per-peer packetization at the edge.

Each EncoderPool holds two kinds of encoders:

Always-on (baseline) encoders

Constructed eagerly at pool creation so any browser can subscribe instantly. The baseline codec (BASELINE_CODEC) is platform-dependent:

• macOS / Linux: VP8 simulcast — up to three layers at full / half / quarter resolution (LayerSpec::vp8_simulcast, RIDs f / h / q). VP8 is the universal codec — Safari, Firefox, Chrome, and Edge all decode it reliably, and it has a long track record on screen content. libvpx has no host dependency that can be absent, so a construction failure at startup is treated as fatal (panic).
• Windows: a single full-resolution H.264 layer via the Media Foundation software encoder. VP8/libvpx is gated off Windows (it needs a C toolchain plus vpx headers), so H.264 — also universally decodable by WebRTC browsers — is the baseline there. H.264 is not simulcast in the pool, so the always-on bank is a single layer. A Windows baseline construction failure degrades (logs and leaves an empty bank, dashboard stays up) rather than panicking, because the pool is built eagerly at --web startup and a panic would take the whole daemon down.

“Always-on” means the encoder threads are spawned eagerly; it does not mean every layer emits frames. Which layers actually encode is governed by a demand-bound and capacity policy. By default:

• a local single-RID viewer demands f only → only the full layer emits;
• a federated single-encoding viewer demands q only → only the quarter layer emits;
• an opt-in multi-RID viewer whose offer carries a=simulcast:recv f;h;q demands all three — the experimental adaptive-bandwidth path.

A paused VP8 encoder thread blocks in blocking_recv at negligible cost; the active layer costs roughly 5% of a core.

On-demand encoders

Spawned when the first peer that needs them joins, torn down when the last such peer leaves. Today: H.264 (and declared but not-yet-wired VP9 / AV1). These exist for browsers that prefer or only support a non-VP8 codec. On-demand encoders are refcounted by viewer count and released deterministically via a PoolLease RAII handle whose Drop decrements the refcount synchronously (so it works from any context, including teardown). A per-slot generation token guards against a stale lease decrementing a replaced slot after a resize.

Layer policy coordinator

A single coordinator task per display (crates/intendant-display/src/aggregator.rs) is the sole owner of pool.pause_layer / pool.resume_layer decisions. Three policies vote, and the coordinator composes their votes by intersection — pause wins; resume requires every active policy to agree:

• Presence policy — pauses all layers after the display sits at zero peers for a 5 s debounce (absorbs browser refreshes and federation reconnect blips), resumes on the first peer. The pool-feed bridge keys its conversion idle gate off the resulting pool state: while the pool has no active consumer (no unpaused always-on layer, no subscribed on-demand encoder) and no peer-join burst window is open, the bridge skips BGRA→I420 conversion entirely — an agent-only session retains only the newest captured frame (cheap Arc stash) instead of converting at capture rate for a stream nobody decodes, and the first tick after a viewer joins converts that retained frame on demand.

  Demand also propagates one stage further upstream, into capture pacing (crates/intendant-display/src/capture/pacing.rs): the session composes a demand probe — encoder-pool consumers, the peer-join burst window, tile subscribers, external raw-frame subscribers, and recent external latest_frame reads or injected input — and installs it on the backend. The X11 and synthetic backends consult it and drop the capture copy itself from configured fps to a 1 fps keepalive while nothing consumes frames, waking within one 50 ms poll slice when demand returns. It is a rate reduction, never a stop: a stopped capture has cold-start latency and breaks latest_frame consumers, while the 1 fps floor keeps latest_frame fresh enough for an instant first paint, a ≤1 s-stale screenshot, and the FrameRegistry’s 1 Hz sampler. ScreenCaptureKit and PipeWire keep their event-driven pacing; the Windows GDI/DXGI backend currently ignores the probe and keeps its own capture rate.

  Per-peer demand is released under tile mode as well: a federated viewer whose client is painting tiles (video element hidden) enters video standby — it stops contributing to the demanded/pinned layer sets, and an on-demand encoder slot pauses once every holder is in standby — so a tile-only session stops paying VP8/H.264 encode while any non-standby viewer on the same display keeps its layers running. The fallback_to_video edge restores demand before the browser is told to switch surfaces, kicks the coordinator for an immediate resume (whose paused→active edge forces a keyframe), and reuses the peer-join keyframe + burst machinery, so the fallback still paints promptly.

• Aggregate-TWCC policy — per-peer cascaded loss. On sustained packet loss it pauses the top layer first, then the middle, reversing on recovery. This is the actionable signal source on the current rtc 0.9 + WKWebView stack.

• Per-RID RR policy — per-(peer, RID) fraction_lost off receiver reports. Currently inert (rtc 0.9 doesn’t populate the RR accumulator) but kept warm for future stacks.

This replaced an earlier design that ran each policy as an independent task writing to the pool, which produced opposite actions when one policy had signal and another defaulted to “wanted.”

PLI coalescing and keyframe-first

When N viewers request a keyframe (PLI/FIR) at nearly the same time, a naive path fires N keyframe requests at the encoder — a 2-3× publisher-side bandwidth amplifier. A KeyframeCoalescer dedupes requests per (codec, rid) within a 50 ms window. Separately, every per-peer forwarder enforces keyframe-first: a peer that joins mid-stream drops P-frames and requests a keyframe until it has seen one, so the decoder never renders garbage.

Per-Peer WebRTC Driver (rtc-rs, sans-I/O)

Each browser connection is a WebRtcPeer (crates/intendant-display/src/webrtc/: the peer handle in mod.rs, ICE-TCP registries in tcp_mux.rs, candidate gathering in ice.rs, offer handling + construction in offer.rs, the driver task in driver.rs, and pool glue in pool_glue.rs) that runs its own tokio driver task. The driver holds a sans-I/O rtc crate (rtc-rs, pinned to =0.9.0) RTCPeerConnection plus its own UDP/TCP sockets, and pumps everything in a single select! loop:

1. inbound UDP/TCP datagrams → peer.handle_read(...)
2. encoded frames from the pool fan-out → packetize → writer.write(...)
3. commands from the public handle (ICE candidates, clipboard, authority state, shutdown) → data-channel writes

After every input the driver drains the connection’s pending writes, reads, and events, and uses poll_timeout / handle_timeout to drive timers.

Because rtc is sans-I/O — it owns no sockets and no clock — the application owns two responsibilities the library would otherwise hide:

• Socket pumping. The driver task is the only code that can write RTP for its peer, which is exactly why per-peer codec/layer selection and packetization live inside the driver rather than in a separate forwarder module (a separate task can’t reach the driver’s RTC state). crates/intendant-display/src/forward.rs is now reduced to SDP/codec-preference helpers.
• TWCC tapping. rtc 0.9 consumes inbound TWCC (transport-wide congestion control) feedback internally and never surfaces it to the application, and its remote-inbound stats accumulator stays at zero. The only place to observe the signal without patching rtc is inside its interceptor chain. crates/intendant-display/src/twcc_tap.rs wires a passive TwccTapInterceptor as the outermost interceptor: it downcasts each inbound RTCP packet, projects a compact event onto a channel, and forwards the original unchanged. A health aggregator turns that event stream into a 1 s TwccHealth snapshot the layer-policy coordinator reads. (The driver also derives a recent send-bitrate estimate from bytes_sent deltas, because rtc 0.9’s available_outgoing_bitrate field is never written.)

Codec negotiation

On offer, the browser’s SDP is parsed for supported codecs. The pool subscribes the peer to every codec it can decode and the driver picks one at frame time; H.264 in particular is matched on its full fmtp identity (profile-level-id + packetization-mode), not just the codec name, because browser negotiation discriminates parameter sets. Each EncodedFrame carries a PayloadSpec so the driver can verify a frame matches the peer-negotiated RTP codec before packetizing. Different peers can land on different payload types for the same codec; the driver rewrites the PT, SSRC, sequence numbers, and timestamps per peer.

ICE-TCP multiplexing

When a browser can’t reach the agent’s UDP candidates — typically because the agent is inside a NAT’d VM with only the dashboard port forwarded — the pipeline falls back to ICE-TCP multiplexed onto the same HTTP port that serves the dashboard, so no extra port-forward is needed.

A shared TcpPeerRegistry is created once at gateway startup. Each peer pre-generates its ICE ufrag and registers it. The gateway’s accept loop peeks the first bytes of every TCP connection to tell HTTP vs. WebSocket vs. STUN-framed traffic apart; STUN-framed connections are read one RFC 4571 frame at a time, the USERNAME’s target-ufrag half is extracted, and the connection is handed to the matching peer’s driver.

The advertised TCP candidate’s IP is derived from the browser’s Host: header — whatever non-loopback IP the browser used to load the dashboard is what the server advertises:

• Via a routable IP (e.g. https://192.168.1.42:8765) — ICE-TCP works automatically.
• Via http://localhost:8765 — ICE-TCP does not work: Firefox and Chrome filter remote loopback candidates as an anti-rebinding mitigation. Bind the port-forward on all interfaces and access via the host’s LAN IP instead.

For VirtualBox NAT users:

VBoxManage modifyvm <vm> --natpf1 delete intendant
VBoxManage modifyvm <vm> --natpf1 "intendant,tcp,0.0.0.0,8765,,8765"

Then access the dashboard at https://<host-LAN-IP>:8765 after enrolling the browser/client certificate with intendant access setup. Use --no-tls --bind 127.0.0.1 only for explicit local/debug plaintext.

Plain HTTP is sufficient for the display transport itself, but not for browser APIs that require a secure context. Use default HTTPS/mTLS, native --tls with a trusted cert, or the macOS app wrapper when the same dashboard session also needs Station WebGPU, microphone/camera, browser screen capture, or stricter clipboard APIs. See Web Dashboard: Secure Browser Contexts.

TURN relay candidates

On a host with no inbound reachability at all (e.g. a cloud container behind NAT with no inbound UDP), even srflx candidates don’t pair. The server-side rtc peer can allocate its own relay on the configured coturn and trickle a typ relay candidate, so media bounces through coturn from both ends. This reuses rtc 0.9’s sans-I/O TURN client and is off the setup critical path: the SDP answer returns with host/srflx/ICE-TCP candidates immediately, and the relay candidate is trickled later only if the allocation succeeds — an unreachable TURN server costs zero added latency.

Tile Streaming (Dirty-Region)

Whole-frame VP8 spends bytes re-encoding the static background even when 90-95% of the frame is unchanged, which fails the visual-freshness bar at full resolution regardless of bitrate. Tile streaming inverts the model — encode only what changed — the way VNC/RFB and its derivatives (Spice, RDP, Citrix HDX) have for decades. The implementation lives in crates/intendant-display/src/tile/; the full design rationale is in docs/design-tile-streaming.md.

Data channels

Three browser-created data channels ride the same RTCPeerConnection as the video track, chosen deliberately rather than “everything reliable”:

The wire format is a versioned little-endian binary framing. Each on-wire message is capped at MAX_DATACHANNEL_MESSAGE_SIZE = 32 KiB (the conservative SCTP datachannel ceiling); snapshots and large tile-updates are chunked, and the browser reassembles by snapshot_id. epoch changes when the world changes shape (resize, fallback transition); seq is monotonic within an epoch and drives a per-tile staleness check.

Tiles, damage, and fallback

• Tiles are 64×64 px (TILE_STREAM_TILE_SIZE_PX).
• Damage comes from platform metadata when possible. On X11, XDamage (crates/intendant-display/src/capture/x11_damage.rs) reports real OS-level dirty rects (ReportLevel::BoundingBox). On macOS, ScreenCaptureKit dirty rect extraction is on by default (INTENDANT_MACOS_SCK_DIRTY_RECTS=0 is the opt-out escape hatch): frames carry SCK’s native dirty rects, a frame whose sample lacks the attachment ships a conservative full-frame rect (the Chromium missing-attachment rule), and the composited cursor (shows_cursor(true)) is content change inside the same pipeline that mints the rects — so the X11 hardware-cursor hazard does not transfer. Other paths use a CPU-bound frame-diff fallback (crates/intendant-display/src/capture/frame_diff.rs) that hashes every tile and emits the ones whose hash changed. A damage backend reporting DamageCapability::None means no platform damage metadata is active; the bridge still runs that CPU frame-diff fallback.
• Tile ↔ video fallback policy (crates/intendant-display/src/tile/policy.rs) switches to whole-frame video on high-motion content and back to tiles when motion subsides, with hysteresis to prevent flapping: enter video at 25% dirty fraction (ENTER_VIDEO_THRESHOLD), exit at 15% (EXIT_VIDEO_THRESHOLD), over a rolling window of 8 samples (HISTORY_K), with a 500 ms minimum dwell (MIN_DWELL) capping switches at 2/sec. The fallback target is the proven baseline video track (VP8-q on macOS/Linux, the full-resolution H.264 layer on Windows), which stays negotiated the whole time; while a viewer’s tile stream is active its video encoders idle in per-peer standby (see the layer-policy section above), and the fallback edge restores demand + forces a keyframe before the surface switch.
• Cursor is sent as a separate CursorState frame on the control channel and drawn as a browser overlay sprite (Path A). X11 typically renders the cursor as a hardware overlay that does not fire XDamage, so dirtying tiles for cursor motion wouldn’t work anyway.

Recovery and backpressure

Recovery is bounded so it can never become the flood it’s recovering from. A periodic snapshot bounds time-to-recover from any silent corruption (30 s in tile mode, 60 s in video-fallback mode). Browser GapReports are satisfied from a bounded per-tile replay buffer (crates/intendant-display/src/tile/recovery.rs) when the whole missing sequence range is still present, otherwise a fresh (rate-limited) snapshot is sent. The initial baseline is also single-source: peer registration sends no snapshot; the browser’s reliable Subscribe control message requests exactly one after its channels are ready. Tile-delta sends are gated by a backpressure monitor (crates/intendant-display/src/tile/backpressure.rs) watching each channel’s buffered amount; the control channel is never throttled, so input and cursor stay prompt under tile pressure.

Per-Platform Backend Matrix

VP8 (crates/intendant-display/src/encode/vp8.rs, libvpx) is the always-on baseline on macOS/Linux and is gated off Windows entirely. See Windows Support for the Windows backends in detail.

DisplayBackend auto-detects the active backend at runtime: on Linux it checks WAYLAND_DISPLAY before falling back to DISPLAY; macOS and Windows always use the native backend.

One deliberate exception: synthetic display mode (crates/intendant-display/src/synthetic.rs), the headless-test-rig backend. When INTENDANT_MOCK_DISPLAY=synthetic is set alongside PROVIDER=mock (and only then — the controller’s gate fails closed, mirroring the mock provider’s own explicit opt-in), display enumeration and every user-display activation are served by a deterministic 1280×720 test card generated by a plain thread, and no native capture API in the matrix above is touched. Synthetic sessions also run with an empty always-on encoder bank (no Media Foundation / libvpx spin-up for a stream nobody watches). This is what keeps the mock-provider e2e suite displayless on all three CI platforms.

Every backend implements the same capture teardown contract (doc-commented on the DisplayBackend trait in crates/intendant-display/src/lib.rs — that doc is canonical): after stop_capture() returns, the frame channel closes within bounded time; no late captured-frame callback may touch freed state (each backend owns whatever quiesce its OS needs — the thread-backed backends join their capture thread, while macOS gates ScreenCaptureKit’s unjoinable callback queue behind a per-session shutdown flag and sender slot, because SCK has been observed delivering frames ~53 s after stop); double-stop and stop-without-start are no-ops; and start-after-stop yields a fresh, clean session. The capture_teardown_contract tests stress the contract on synthetic backends in CI, and each real OS backend has an #[ignore]d start/stop stress test for operator hardware (cargo test -p intendant-display --lib -- --ignored real_capture_stress).

Remote Input Path

Browser keyboard/mouse events (kd/ku/md/mu/mm/sc — key and mouse button edges, absolute mouse moves, relative scroll deltas) reach the daemon on three lanes: authenticated peers’ WebRTC data channels (control/pointer, reliable and ordered), the daemon-origin dashboard-control tunnel used by trusted local or direct-mTLS sessions (plus local development builds of the packaged macOS bridge), and the legacy /ws gateway socket on a trusted daemon surface. All three converge on a per-display-session ordered input queue (crates/intendant-display/src/input_queue.rs): each lane’s authority gate runs first (a refused event never enqueues), the queued event retains that live authorization predicate, and the pump rechecks it immediately before injection so revocation discards buffered input. The enqueue is a sync non-blocking push (the two WebRTC lanes push from inside sans-I/O rtc poll loops that must never block), and a single pump task per session drains the queue, awaiting each injection to completion before the next — so injection order always equals wire arrival order. (The pre-queue design dispatched one tokio::spawn per event, which could reorder adjacent events under load — a kd/ku inversion presents as a stuck auto-repeating key.)

The queue coalesces adjacent pending mouse-moves latest-wins (both carry absolute coordinates; an mm never jumps over a discrete event, preserving position-at-click semantics). Scroll events are never coalesced — their deltas are relative, and merging would change total scroll distance. On overflow (soft cap 256) the oldest continuous event (mm/sc) is evicted first; discrete events are never sacrificed for continuous ones — an all-discrete backlog instead grows to a hard cap (1024). Reaching that cap trips the browser-input lane as a unit: pending events are cleared, no further browser events are accepted for that display session, and the pump attempts a synthetic key-up/mouse-up for every edge it pessimistically tracks as held. That terminal trip is reserved for the hard-cap invariant violation. Ordinary authority changes, source transport closure, and queued events whose live authorization revision has expired take a recoverable reset: discard the old epoch’s backlog, synthesize releases for its held edges, then let the next authorized source use a fresh queue generation. A dashboard-control handoff overflow similarly cancels that overloaded connection after resetting its source; it does not poison the display session for a later connection. No path drops one key/button edge and silently continues the same authority epoch. Browser blur/pointer-cancel handling sends the same releases as defense in depth; server-side tracking remains authoritative for abrupt disconnects. Computer-use actions bypass the queue via DisplaySession::inject_input, which awaits completion (a CU click must be injected before the follow-up screenshot).

Browser input is physical-key-only (Phase 1). Injected key events use the DOM code field (physical key position), not key. Non-US keyboard layouts therefore produce incorrect character output until a future phase adds character-level injection. Browser and server teardown reset every tracked key and mouse button, not only modifiers.

Bidirectional Clipboard

The ClipboardMonitor (crates/intendant-display/src/clipboard.rs) polls the system clipboard every 500 ms and syncs changes over a WebRTC data channel in both directions. It supports text and images (PNG, capped at 5 MB). On copy in the browser, content is pushed to the display’s clipboard; on copy in the display, content is pushed to the browser. A DisplayView grant alone does not enable that channel: both inbound mutation and outbound clipboard content require the same live DisplayInput authority as interactive keyboard/mouse control, rechecked again when a queued clipboard command is sent. The per-platform read/write tools are listed in the matrix above. Polling is gated on viewer presence: with zero connected peers each tick is a no-op (no pbpaste/osascript/xclip/wl-paste subprocess is spawned for a sync nobody receives), and content that changed during the pause is re-baselined silently on resume rather than replayed to the next viewer.

Multi-Monitor

Each display gets a stable display_id (0 is always the primary). DisplayInfo carries the intendant-stable id plus the native platform_id (CGDisplayID on macOS, X11 screen number, PipeWire node_id on Wayland, DXGI output on Windows). The pipeline supports enumeration, a dashboard display picker, per-display metrics, dynamic resize (encoders are transparently recreated at the new dimensions via the pool’s layer factory, with a DisplayResize event emitted), and hotplug.

On Wayland, the portal does not expose true enumeration before a session is opened, so enumerate_displays returns a placeholder; enumerate_displays_with_sessions patches each entry with the live capture resolution from the session registry so the agent’s click coordinates match the screenshot it receives.

Recording

Display recording runs in parallel with WebRTC streaming, with ffmpeg strictly as the encoder (src/bin/caller/recording.rs). When a DisplaySession exists, frames are polled from the session, JPEG-encoded, and piped to ffmpeg via image2pipe. Without a session (--record-display, the debug screen), Linux lets ffmpeg capture directly via x11grab, while macOS captures the requested display in-process via ScreenCaptureKit and bridges the frames into the same image2pipe encoder — so the Intendant process itself must hold the Screen Recording permission. Recordings are segmented into MP4 files (default 60 s) for efficient seeking; the dashboard provides a player with timeline, seeking, and speed control.

[recording]
enabled = true
framerate = 15
segment_duration_secs = 60
quality = "medium"   # "low" (CRF 35), "medium" (CRF 28), "high" (CRF 20)

Frame Registry

The FrameRegistry stores high-quality JPEG frames captured from displays and browser cameras in the session directory (<session>/frames/, metadata in frames.jsonl). Frames serve two purposes: context for CU models (auto_attach_display_frames() grabs the latest frame per stream for the agent’s next turn) and presence inspection (inspect_frame / inspect_frames tools). Each frame carries a sent_to_live flag so the browser-side live model never sees a frame twice.

WebRTC Configuration

STUN/TURN servers are configured via [webrtc] in intendant.toml (empty by default — local-only, no STUN/TURN):

[webrtc]
federation_allow_h264 = false   # see Peer Federation; VP8-pinned over relays by default

[[webrtc.ice_servers]]
urls = ["stun:stun.l.google.com:19302"]

[[webrtc.ice_servers]]
urls = ["turn:turn.example.com:3478"]
username = "user"
credential = "pass"

federation_allow_h264 governs only the cross-machine federated path (see Peer Federation); the local same-machine display path is unaffected.

Display Metrics

Per-display atomic counters feed the dashboard Stats tab: capture FPS, encode FPS, encode freshness (how stale emitted frames are at output time — not encoder speed; an idle desktop driven only by the periodic snapshot can show seconds), capture/encode/peer drop counts, peer count, resolution, and a full set of tile-stream counters (dirty rects, dirty tiles, delta/snapshot FPS and kbps). Rates are computed over the elapsed window and counters reset on read.

Known Limitations

• Physical-key-only input breaks non-US keyboard layouts (Phase 1).
• Tile streaming still depends on data-channel viewers. X11 uses XDamage; macOS uses ScreenCaptureKit dirty rects by default (INTENDANT_MACOS_SCK_DIRTY_RECTS=0 opts out to frame-diff); Wayland currently uses the CPU-bound frame-diff path.
• Wayland enumeration is portal-limited — true multi-monitor identity before a session opens is not available.
• rtc 0.9 doesn’t surface TWCC or populate RR stats, hence the interceptor tap and the bytes_sent-delta bitrate estimate; per-RID RR-driven layer policy is inert on this stack.
• No virtual-display equivalent on macOS or Windows — capture targets the real session only. macOS can expose a single real native window as a capture target, but that window still belongs to the user’s logged-in desktop session.

See Also

• Peer Federation — sharing a remote peer’s display across machines, LAN/mTLS, and the [webrtc] federation knobs
• Windows Support — the Windows capture/input/encode backends in detail
• Computer Use & Live Audio — input and voice