databendlabs/databend · error

hybrid bitmap small set size overflow

Error message

hybrid bitmap small set size overflow: {}

What it means

HybridBitmap's Small variant stores at most u8::MAX (255) values. serialize_into writes the set length as one byte; if the small set holds more than 255 elements the u8::try_from conversion fails and serialization aborts with this InvalidData io error instead of silently truncating.

Solutions

1. Check how the HybridBitmap was constructed; sets over 255 elements must use the large representation.
2. Ensure inserts go through the library's APIs that promote Small to Large when len exceeds 255.
3. If you hit this via aggregation/grouping code, report it — it indicates an internal invariant bug.

if bitmap.len() > 255 {
    return Err(anyhow!("small-set bitmap exceeds 255 elements"));
}

When it happens

Trigger: Calling serialize_into on a HybridBitmap::Small whose set length is 256 or greater — an internal invariant violation, since larger sets should have been promoted to the large/roaring representation.

Common situations: A bug or manual construction that inserts more than 255 values into the Small variant, bypassing the size-based promotion logic.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• internal error: entered unreachable code
• Unimplemented serialize DataSourceMeta
• expected a non-NaN float
• data is truncated or invalid
• index out of range

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/70f883329ba81fe5. Report an issue: GitHub.

Appendix: source

Thrown at src/common/io/src/bitmap.rs:216

                rhs.iter().filter(|v| lhs.contains(**v)).count() as u64
            }
            (HybridBitmap::Small(lhs), HybridBitmap::Large(rhs)) => {
                lhs.iter().filter(|v| rhs.contains(**v)).count() as u64
            }
            (HybridBitmap::Small(lhs), HybridBitmap::Small(rhs)) => {
                small_intersection_len(lhs, rhs)
            }
        }
    }

    pub fn serialize_into<W: io::Write>(&self, mut writer: W) -> io::Result<()> {
        writer.write_all(&HYBRID_MAGIC)?;
        writer.write_all(&[HYBRID_VERSION])?;
        match self {
            HybridBitmap::Small(set) => {
                writer.write_all(&[HYBRID_KIND_SMALL])?;
                let len = u8::try_from(set.len()).map_err(|_| {
                    io::Error::new(
                        io::ErrorKind::InvalidData,
                        format!("hybrid bitmap small set size overflow: {}", set.len()),
                    )
                })?;
                writer.write_all(&[len])?;
                for value in set.iter() {
                    writer.write_all(&value.to_le_bytes())?;
                }
                Ok(())
            }
            HybridBitmap::Large(tree) => {
                writer.write_all(&[HYBRID_KIND_LARGE])?;
                tree.serialize_into(writer)
            }
        }
    }

    pub fn iter(&self) -> HybridBitmapIter<'_> {

View on GitHub (pinned to 288d84d76e)