niri-wm/niri · error

unsupported shm format

Error message

unsupported shm format: {format:?}

What it means

render_to_shm only accepts wl_shm buffers whose pixel format is Xrgb8888 or Argb8888; any other wl_shm::Format (e.g. Xbgr8888, RGB565, or 10-bit formats) hits the catch-all arm and bails. The conversion to a smithay Fourcc simply has no mapping for other formats, so rendering into that SHM buffer is refused early.

Solutions

1. Reallocate or request the target buffer with wl_shm::Format::Xrgb8888 or Argb8888 before calling render_to_shm
2. If you control buffer negotiation, restrict the advertised shm formats to those two
3. Add a Fourcc mapping for the needed format in the match and confirm the byte layout matches the little-endian assumption

Example fix

// before
let attrs = BufferAllocation { format: wl_shm::Format::Xbgr8888, .. };
// after
let attrs = BufferAllocation { format: wl_shm::Format::Argb8888, .. };

if !matches!(format, wl_shm::Format::Xrgb8888 | wl_shm::Format::Argb8888) {
    return Err(anyhow!("capture buffer must be Xrgb8888/Argb8888, got {format:?}"));
}

fn is_supported_shm_format(f: wl_shm::Format) -> bool {
    matches!(f, wl_shm::Format::Xrgb8888 | wl_shm::Format::Argb8888)
}

match render_for_image_copy_capture(...) {
    Err(e) if e.to_string().contains("unsupported shm format") => {
        // reallocate buffer as Argb8888 and retry once
    }
    other => other?,
}

When it happens

Trigger: Calling render_for_image_copy_capture, render_for_image_copy_cursor_capture, or render_for_screencopy_internal with a target wl_shm buffer whose shm::Format is not Xrgb8888/Argb8888 — e.g. a client or capture pipeline negotiating a different shm format.

Common situations: Screencast/capture clients negotiating non-8-bit or non-BGRA shm formats; copy-pasted buffer allocation code using a different wl_shm::Format; protocol peers advertising formats the renderer never intended to support.

Related errors

• invalid buffer format or size
• short texture mapping
• buffer does not fit in its shm pool

AI-assisted analysis of niri-wm/niri@62c230a662 (2026-09-19). Data as JSON: /api/errors/ae1b3a6bf866bc16. Report an issue: GitHub.

Appendix: source

Thrown at src/render_helpers/mod.rs:345

}

pub fn render_to_shm(
    renderer: &mut GlesRenderer,
    damage_tracker: &mut OutputDamageTracker,
    buffer: &WlBuffer,
    format: wl_shm::Format,
    elements: &[impl RenderElement<GlesRenderer>],
    states: RenderElementStates,
) -> anyhow::Result<()> {
    let _span = tracy_client::span!();
    // The pointer and length are the client's entire pool, which may hold other
    // buffers besides this one... buffer_data is the one we want.
    shm::with_buffer_contents_mut(buffer, |pool, pool_len, buffer_data| {
        let (size, _scale, _transform) = damage_tracker.mode().try_into().unwrap();
        let fourcc = match format {
            wl_shm::Format::Xrgb8888 => Fourcc::Xrgb8888,
            wl_shm::Format::Argb8888 => Fourcc::Argb8888,
            _ => bail!("unsupported shm format: {format:?}"),
        };

        ensure!(
            // The buffer prefers pixels in little endian ...
            buffer_data.format == format
                && buffer_data.width == size.w
                && buffer_data.height == size.h,
            "invalid buffer format or size"
        );

        // The client chooses the stride and may pad rows, so only the first
        // row_len bytes can be used here.
        let row_len = size.w as usize * 4;
        let height = size.h as usize;
        let offset = usize::try_from(buffer_data.offset).context("negative buffer offset")?;
        let stride = usize::try_from(buffer_data.stride).context("negative buffer stride")?;

        // This should have already been validated by wl_shm, and a pool can

View on GitHub (pinned to 62c230a662)