databendlabs/databend · error

data is truncated or invalid

Error message

data is truncated or invalid

What it means

BitmapReader::decode validates that the input buffer is at least as long as the container area requires (computed from the last container's cardinality, or BITMAP_BYTES for array containers). If buf is shorter, the encoded bitmap data is incomplete or corrupt and decoding fails with UnexpectedEof.

Solutions

1. Verify the full serialized bitmap bytes were written/read (check write completeness and file sizes).
2. Confirm the byte source matches the expected bitmap encoding/version (check magic/version bytes upstream).
3. Re-export or recompute the bitmap data if the payload is confirmed corrupt.

Example fix

// before
let reader = BitmapReader::decode(&partial_bytes)?;

// after
if partial_bytes.len() < expected_min_size {
    return Err(anyhow!("bitmap payload truncated: {} < {}", partial_bytes.len(), expected_min_size));
}
let reader = BitmapReader::decode(&partial_bytes)?;

if bytes.len() < expected_min_bitmap_size {
    return Err(anyhow!("bitmap payload too short"));
}

match BitmapReader::decode(&bytes) {
    Ok(r) => r,
    Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => {
        log::warn!("truncated bitmap payload"); return Err(e.into());
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Calling BitmapReader::decode (e.g. via Bitmap deserialization) with a byte slice that was truncated in transit, sliced incorrectly, or produced by an incompatible encoder.

Common situations: Network/storage truncation of serialized bitmaps; passing a header-only slice instead of the full payload; reading a file written by a different bitmap format version.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• container offset exceeds bitmap data
• Unimplemented deserialize DataSourceMeta
• expected a non-NaN float
• hybrid bitmap small set size overflow
• index out of range

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/fa6d15c353f3494b. Report an issue: GitHub.

Appendix: source

Thrown at src/common/io/src/bitmap/reader.rs:222

        }

        let last_container = (containers - 1) as i64;
        reader.seek_relative(last_container * DESCRIPTION_BYTES as i64 + 2)?;
        let last_cardinality = reader.read_u16::<LittleEndian>()? as usize + 1;

        reader.seek_relative(last_container * OFFSET_BYTES as i64)?;
        let last_offset = reader.read_u32::<LittleEndian>()?;

        let size = 4
            + last_offset as usize
            + if last_cardinality <= ARRAY_LIMIT {
                2 * last_cardinality
            } else {
                BITMAP_BYTES
            };

        if buf.len() < size {
            Err(Error::new(
                ErrorKind::UnexpectedEof,
                "data is truncated or invalid",
            ))
        } else {
            Ok(BitmapReader {
                prefix,
                containers,
                buf: &buf[..size],
            })
        }
    }

    pub fn containers(&self) -> usize {
        self.containers as usize
    }

    pub fn prefix(&self) -> u32 {
        self.prefix

View on GitHub (pinned to 288d84d76e)