quickwit-oss/quickwit · error · anyhow::Error

data too short (len= )

Error message

data too short (len={})

What it means

HotDirectory caches split footer/metadata in a serialized blob: a 4-byte little-endian length prefix followed by a postcard-serialized `HotDirectoryMeta`. `deserialize_impl` bails with "data too short" when fewer than 4 bytes remain, because it cannot even read the length prefix.

Solutions

1. Re-upload or re-fetch the split: the hotcache blob is truncated, so delete and re-index the affected split.
2. Verify the bytes passed to deserialize are the hotcache file contents (e.g. `.hotcache`) and not another artifact.
3. Check storage/network for partial reads; confirm the object's content length matches the manifest.

fn hotcache_bytes_look_valid(bytes: &[u8]) -> bool { bytes.len() >= 4 }

match hot_directory::deserialize(bytes) {
    Ok(meta) => Ok(meta),
    Err(e) if e.to_string().contains("data too short") => {
        // treat split as corrupted: fail leaf search for this split and request re-index
        Err(anyhow::anyhow!("corrupt/empty hotcache: {e:#}"))
    }
    Err(e) => Err(e),
}

When it happens

Trigger: Calling `HotDirectory::deserialize`/open with an `OwnedBytes` shorter than 4 bytes — e.g. a truncated or empty hotcache file read from storage, or feeding the deserializer the wrong (non-hotcache) bytes.

Common situations: Corrupted or partially uploaded split where the hotcache byte array was truncated; storage returning empty object content on failed read; hotcache written by an incompatible/older writer producing zero-length payload.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• `append_records` should be called with `position_opt: None`
• bundled file range overlaps split footer
• bundled file range starts after it ends
• `doc_batch` should not be empty
• dynamic message: storage_err.source.to_string() converted…

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/2ce6a217b83dfa34. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-directories/src/hot_directory.rs:58

    const MAGIC_NUMBER: u32 = 2_557_869_106u32;
    type Component = HotDirectoryMeta;

    fn to_version_code(self) -> u32 {
        self as u32
    }

    fn try_from_version_code_impl(code: u32) -> Option<Self> {
        match code {
            1u32 => Some(Self::V1),
            _ => None,
        }
    }

    fn deserialize_impl(&self, bytes: &mut OwnedBytes) -> anyhow::Result<HotDirectoryMeta> {
        match self {
            Self::V1 => {
                if bytes.len() < 4 {
                    bail!("data too short (len={})", bytes.len());
                }
                let len = bytes.read_u32() as usize;
                let hot_directory_meta = postcard::from_bytes(&bytes.as_slice()[..len])
                    .context("failed to deserialize hot directory meta")?;
                bytes.advance(len);
                Ok(hot_directory_meta)
            }
        }
    }

    fn serialize_impl(component: &Self::Component, output: &mut Vec<u8>) {
        let buf = postcard::to_stdvec(component).unwrap();
        output.extend_from_slice(&(buf.len() as u32).to_le_bytes());
        output.extend_from_slice(&buf[..]);
    }
}

#[derive(Serialize, Deserialize)]

View on GitHub (pinned to a39730c5cd)