Back to Blog

Wayland Screen Capture for a Remote-Desktop Agent with XDG Portals, PipeWire, and x11vnc

Christopher 6 min read
linuxwaylandremote-desktoppipewire

This post documents how the ET Ducky Linux agent captures the screen on Wayland desktops. It covers the xdg-desktop-portal ScreenCast/RemoteDesktop flow over D-Bus, the PipeWire-to-GStreamer pipeline that encodes the stream, and how the helper process gets into the user's session from a system service. It also covers why the portal refuses to persist input permission and where the x11vnc path is still used as a fallback. Everything here is the shipped implementation, a single self-contained Python helper plus the agent code that spawns it.

Why the x11vnc path was replaced

The first-generation Linux capture path ran x11vnc against the logged-in user's X display. It works against real Xorg. It fails on XWayland's default visuals with BadMatch on XGetImage, and Ubuntu 22.04+, Fedora, and RHEL 9+ desktops default to GNOME on Wayland. The replacement targets the freedesktop portal API instead of accumulating per-compositor capture code. GNOME, KDE, and wlroots compositors all ship an xdg-desktop-portal backend, the portal owns the permission UX, and Wayland and X11 sessions produce identical output through one code path.

Getting into the session

The structural problem for any agent is that xdg-desktop-portal is a per-user D-Bus service, and PipeWire screen capture requires permission from the active session's compositor. A system service cannot call it. The agent (running as an unprivileged etducky system user) therefore spawns the helper into the logged-in user's session with sudo -n -u <user>, backed by a narrow sudoers entry, and reads frames back over a Unix socket.

Two details cost real debugging time. First, the socket is an abstract socket, with a leading NUL byte and no filesystem presence. The systemd sandboxing directives on the agent unit, PrivateTmp and BindPaths, apply to filesystem paths, and an abstract name avoids that entire class of failure. Second, sudo is invoked without -i. A login shell resets the environment and can leave the helper without a usable D-Bus session address. Direct exec plus the helper's own fallback to /run/user/<uid>/bus, the systemd-conventional path libdbus also probes, connects reliably.

The agent's selection logic tries the helper first rather than probing for portal support. If a user is logged in and the helper binary exists, the agent spawns it and gives it fifteen seconds to bind the socket. On failure the agent falls back to x11vnc. Testing for portal availability up front would duplicate the helper's own startup logic. The helper is the probe, and its exit codes (1 portal denied, 2 D-Bus failed, 3 PipeWire failed, 4 GStreamer failed, 5 socket bind failed) make each failure mode legible in the journal.

The portal flow, step by step

The helper drives two portal interfaces: org.freedesktop.portal.ScreenCast and org.freedesktop.portal.RemoteDesktop. The ordering rules are strict and mostly undocumented outside the spec:

  1. CreateSession runs on the RemoteDesktop interface whenever input is needed. Input cannot be added to an existing ScreenCast-only session. Combining capture and input means creating the session on RemoteDesktop and layering ScreenCast onto it. On compositors without a RemoteDesktop backend, CreateSession returns NotSupported and the helper falls back to a ScreenCast-only session. Capture works, input calls become silent no-ops, and the dashboard shows a view-only session.
  2. SelectDevices (RemoteDesktop only) requests keyboard and pointer. It must happen between CreateSession and SelectSources or input is not granted at Start.
  3. SelectSources requests monitor capture with the cursor embedded in the stream. This call also carries the persistence setting, covered below.
  4. Start returns the PipeWire stream node IDs and the granted input-device bitmask. A RemoteDesktop session that returns zero granted devices means the operator declined input in the dialog. The helper degrades to view-only instead of failing.
  5. OpenPipeWireRemote returns a file descriptor for the PipeWire connection over D-Bus fd-passing. Capturing by portal-issued fd, instead of connecting to the daemon by name, makes the portal the authority on which PipeWire instance to use.

Portal request and response correlation works by constructing an expected request-object path from the caller's bus name and a caller-chosen token, subscribing to its Response signal, then issuing the call with a 30-second timeout. The timeout is that long because the user has to answer a permission dialog first. Some portal versions return a different request handle than the constructed one. The helper re-subscribes to the actual handle and waits again, which accommodates that version drift.

Persistence and why the portal does not allow unattended input

The portal's persistence setting (persist_mode) lets a user's sharing choice be remembered across sessions. The shipped configuration differs between the two session types:

A Wayland agent cannot do fully unattended remote control through the portal the way X11 tools can. A person in the session approves input each time. When that requirement does not fit the deployment, such as headless boxes or lights-out maintenance, the remaining options are the x11vnc path on Xorg systems or out-of-band management below the OS.

The encoding pipeline

Frames flow through a GStreamer pipeline the helper builds from the portal-issued fd:

pipewiresrc fd=<fd> path=<node> do-timestamp=true
  ! videoconvert
  ! videorate ! video/x-raw,framerate=<max-fps>/1
  <encoder chain>
  ! appsink emit-signals=true sync=false max-buffers=2 drop=true

The ordering in that pipeline matters. Putting a framerate caps filter directly after the source breaks negotiation, because the compositor decides the actual capture rate. videorate downstream is the correct rate-capping mechanism. The appsink's max-buffers=2 drop=true keeps the pipeline real-time, so a slow consumer drops frames instead of building latency.

Encoder selection probes GStreamer element factories in priority order: vah264enc (VA-API hardware, roughly 5% CPU and under 500 KB/s for 1080p), then x264enc (software, ~30% CPU, ubiquitous), then jpegenc as the legacy fallback (~20% CPU and 5+ MB/s, always available, an order of magnitude more bandwidth). Both H.264 encoders are constrained to baseline profile (avc1.42E01E) because VA-API's default is main and the browser WebCodecs decoders in Safari and older Firefox are stricter than the one in Chrome. Latency tuning is zerolatency/ultrafast on x264 and low-power plus CBR on VA-API, targeting under one frame of encoder buffering. The H.264 path emits a one-shot SPS+PPS config frame before the first NALU. Decoding on the browser side is covered in the WebCodecs post.

Input injection

When the RemoteDesktop grant succeeded, input goes back through the same portal session: NotifyPointerMotionAbsolute, NotifyPointerButton, NotifyPointerAxis, and NotifyKeyboardKeysym. Absolute pointer coordinates are stream-relative, so the pipeline reports the negotiated stream dimensions back to the portal layer. Without the real width and height, absolute motion lands in the wrong place. Each injection call is fire-and-log. A failed injection degrades the session and does not end it.

How the helper is deployed

The helper ships as one dependency-light Python file under /opt/etducky/agent/bin/, using only PyGObject and GStreamer packages present in Ubuntu and Fedora default repositories. There is no virtualenv, no pip, and no compiled extension, so installing the agent package is all that is required to run it. The frame contract on the Unix socket is length-prefixed binary (JPEG frames, H.264 NALUs with a keyframe flag, and the H.264 config record), so the agent side stays codec-agnostic and the same relay handles every encoder the helper might have chosen.

ET Ducky

Documentation and pricing are available on this site.

View Pricing