databendlabs/databend · error · InvalidData

e

Error message

e

What it means

During metadata compression with MetaCompression::Snappy when the 'dev' feature is disabled, compress() deliberately returns ErrorCode::UnknownFormat('unsupported compression: {:?}') because the Snappy path is compiled out. Any caller (RawBlockMeta, encode_column_hll) writing table metadata compressed with Snappy in a non-dev build will fail. It is a build-feature/config mismatch, not a data corruption issue.

Solutions

1. Rebuild the binary with the 'dev' feature enabled so the Snappy encoder is compiled in
2. Rewrite existing metadata with MetaCompression::Zstd or MetaCompression::None using a dev build, then use those in production
3. Find and fix the code/config path that selects MetaCompression::Snappy in non-dev deployments

Example fix

// before
MetaCompression::Snappy
// after
MetaCompression::Zstd  // supported in all builds

if !cfg!(feature = "dev") && *compression == MetaCompression::Snappy {
    return Err(ErrorCode::UnknownFormat("snappy meta compression requires feature 'dev'".into()));
}

fn is_snappy(c: &MetaCompression) -> bool { matches!(c, MetaCompression::Snappy) }

match compress(encoding, &data) {
    Err(e) if e.message().contains("unsupported compression") => fallback_to_zstd(&data),
    other => other,
}

When it happens

Trigger: Calling compress() with compression=MetaCompression::Snappy in a binary built without the 'dev' feature (the #[cfg(not(feature = "dev"))] fallback arm). Any read path that persisted Snappy-compressed meta with a dev build and is then consumed by a non-dev build.

Common situations: Mixing dev and release builds against the same table data; a stored meta file whose header records Snappy while the running binary lacks the feature; manually setting MetaCompression::Snappy in config/code of a production build.

Related errors

• internal error: entered unreachable code
• Unimplemented serialize DataSourceMeta
• expected a non-NaN float
• hybrid bitmap small set size overflow
• fail to serialize Scalar

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/3843c7445a0e4e69. Report an issue: GitHub.

Appendix: source

Thrown at src/query/storages/common/table_meta/src/meta/format.rs:77

                "unsupported compression: {}",
                other
            ))),
        }
    }
}

pub fn compress(compression: &MetaCompression, data: Vec<u8>) -> Result<Vec<u8>> {
    match compression {
        MetaCompression::None => Ok(data),
        MetaCompression::Zstd => {
            let mut encoder = ZstdEncoder::new(Vec::new(), 0)?;
            encoder.write_all(&data)?;
            Ok(encoder.finish()?)
        }
        #[cfg(feature = "dev")]
        MetaCompression::Snappy => Ok(SnapEncoder::new()
            .compress_vec(&data)
            .map_err(|e| Error::new(ErrorKind::InvalidData, e))?),
        #[cfg(not(feature = "dev"))]
        _ => Err(ErrorCode::UnknownFormat(format!(
            "unsupported compression: {:?}",
            compression
        ))),
    }
}

pub fn decompress(compression: &MetaCompression, data: Vec<u8>) -> Result<Vec<u8>> {
    match compression {
        MetaCompression::None => Ok(data),
        MetaCompression::Zstd => {
            let mut decoder = ZstdDecoder::new(&data[..])?;
            let mut decompressed_data = Vec::new();
            decoder
                .read_to_end(&mut decompressed_data)
                .map_err(|e| Error::new(ErrorKind::InvalidData, e))?;
            Ok(decompressed_data)

View on GitHub (pinned to 288d84d76e)