niri-wm/niri · error

buffer does not fit in its shm pool

Error message

buffer does not fit in its shm pool

What it means

render_to_shm re-validates that the source wl_shm buffer's declared layout (offset, stride, height, row length) actually fits within the pool's mapped length. Although wl_shm protocol should have rejected oversized buffers and pools can only grow, this is a defensive internal invariant check before creating a texture from the pool memory. It fires when the buffer's computed end address exceeds pool_len, or when stride is smaller than the per-row byte length.

Source

Thrown at src/render_helpers/mod.rs:335

                && buffer_data.width == size.w
                && buffer_data.height == size.h,
            "invalid buffer format or size"
        );

        // The client chooses the stride and may pad rows, so only the first
        // row_len bytes can be used here.
        let row_len = size.w as usize * 4;
        let height = size.h as usize;
        let offset = usize::try_from(buffer_data.offset).context("negative buffer offset")?;
        let stride = usize::try_from(buffer_data.stride).context("negative buffer stride")?;

        // This should have already been validated by wl_shm, and a pool can
        // only grow, but check again just in case.
        let end = stride
            .checked_mul(height.saturating_sub(1))
            .and_then(|len| len.checked_add(row_len))
            .and_then(|len| len.checked_add(offset));
        ensure!(
            stride >= row_len && end.is_some_and(|end| end <= pool_len),
            "buffer does not fit in its shm pool"
        );

        let mut texture =
            create_texture(renderer, size, fourcc).context("error creating texture")?;
        let mut target = renderer
            .bind(&mut texture)
            .context("error binding texture")?;

        let _res = damage_tracker
            .render_output_with_states(
                renderer,
                &mut target,
                0,
                elements,
                Color32F::TRANSPARENT,
                states,

View on GitHub (pinned to 9e72e4917c)

Solutions

1. Fix the client to size its wl_shm_pool to at least offset + stride * (height - 1) + width * bytes_per_pixel for the chosen fourcc format before attaching the buffer.
2. Verify the buffer's stride is at least width * bytes-per-pixel for the advertised format (e.g. >= width*4 for XRGB8888/ARGB8888).
3. Re-check that height/width/stride/offset passed into render_to_shm come from the wl_buffer, not stale or hand-computed values.
4. Ensure pool resize (wl_shm_pool.resize) is called before the buffer is rendered, not after metadata validation.

Example fix

// before: pool too small for declared geometry
let pool = shm.create_pool(fd, (height - 1) * stride as i32);
let buffer = pool.create_buffer(offset, width, height, stride, format);

// after: pool sized to full buffer extent
let row_len = width * 4; // bytes per pixel for the format
let size = offset + stride * (height - 1) + row_len;
let pool = shm.create_pool(fd, size as i32);
let buffer = pool.create_buffer(offset, width, height, stride, format);

// Validate before calling render_to_shm
fn shm_buffer_fits(pool_len: usize, offset: usize, stride: usize, row_len: usize, height: usize) -> bool {
    if stride < row_len { return false; }
    stride
        .checked_mul(height.saturating_sub(1))
        .and_then(|len| len.checked_add(row_len))
        .and_then(|len| len.checked_add(offset))
        .is_some_and(|end| end <= pool_len)
}
if !shm_buffer_fits(pool_len, offset, stride, row_len, height) {
    return Err("shm buffer exceeds pool size");
}

fn fits_in_pool(end: Option<usize>, pool_len: usize) -> bool {
    end.is_some_and(|end| end <= pool_len)
}

When it happens

Trigger: Calling render_to_shm (directly or via render_for_screencopy_internal) with an shm buffer whose offset + stride*(height-1) + row_len exceeds the pool's current size, or whose stride < row_len (row_len is derived from the fourcc format's bytes-per-pixel times width).

Common situations: A screencopy/compositor client attaching a wl_buffer whose wl_shm_pool was resized after buffer creation metadata was computed; buggy clients advertising a height/width/stride inconsistent with the pool they mapped; format mismatches (e.g. claiming XRGB8888, 4 bytes/px, but computing stride from a smaller bpp); race conditions where pool shrink logic (invalid per protocol) is attempted.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• short texture mapping
• hdisplay {} must be < hsync_start {}
• no X11 support

AI-assisted analysis of niri-wm/niri@9e72e4917c (2026-09-12). Data as JSON: /api/errors/0c9db959e5b72484. Report an issue: GitHub.