qdrant/qdrant · error · io::Error

InvalidData

InvalidData

Error message

Invalid header

What it means

UniversalHashMap::open (the universal-io variant of the persisted hash map) failed to parse the fixed-size Header from the first bytes of the file. Header::read_from_prefix returns an error when there are fewer bytes than size_of::<Header>() or the bytes cannot form a valid Header — in practice an empty, truncated, or non-hashmap file.

Source

Thrown at lib/common/common/src/persisted_hashmap/uio/mod.rs:75

        options.populate = options.populate.or_partial(0..HEADER_AND_BASIC_PHF_SIZE);

        fs.schedule_prefetch(path.as_ref(), Some(options), None)
    }

    /// Load the hash map from file.
    pub fn open<Fs: UniversalReadFs<File = S>>(
        fs: &Fs,
        path: impl AsRef<Path>,
        options: OpenOptions,
        extra: Fs::OpenExtra,
    ) -> UioResult<Self> {
        let storage = TypedStorage::<S, u8>::open(fs, path, options, extra)?;

        // 1. Read header.
        let header_bytes =
            storage.read(ReadRange::new(0, size_of::<Header>() as u64), Sequential)?;
        let (header, _) = Header::read_from_prefix(&header_bytes)
            .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "Invalid header"))?;

        if header.key_type != K::NAME {
            return Err(UniversalIoError::from(io::Error::new(
                io::ErrorKind::InvalidData,
                "Key type mismatch",
            )));
        }

        // 2. Read PHF. The region between the header and buckets_pos contains the
        //    serialised PHF followed by padding; `Function::read` consumes only what
        //    it needs and ignores trailing bytes.
        let phf_region_start = size_of::<Header>() as u64;
        let phf_region_len = header
            .buckets_pos
            .checked_sub(phf_region_start)
            .ok_or_else(|| {
                io::Error::new(io::ErrorKind::InvalidData, "buckets_pos before header end")
            })?;

View on GitHub (pinned to 74f3e85b94)

Solutions

  1. Check the file length is at least the header size and clearly non-zero before opening
  2. Re-download or regenerate the hash map file
  3. Verify the path points at a file actually produced by the persisted-hashmap writer

Example fix

// before
let map = UniversalHashMap::open(&fs, &path, options, extra)?;

// after: guard against empty/partial files
let len = std::fs::metadata(&path)?.len();
if len < MIN_EXPECTED_MAP_SIZE {
    return Err(format!("map file too small: {len} bytes").into());
}
let map = UniversalHashMap::open(&fs, &path, options, extra)?;
Defensive patterns

Strategy: validation

Validate before calling

let len = fs.len(&path)?; // or std::fs::metadata(&path)?.len()
if len < HEADER_AND_BASIC_PHF_SIZE {
    return Err(UniversalIoError::Io(io::Error::new(
        io::ErrorKind::InvalidData,
        format!("map file too small ({len} bytes); refusing to open"),
    )));
}

Try / catch

match UniversalHashMap::open(&fs, &path, options, extra) {
    Err(UniversalIoError::Io(ref e)) if e.kind() == io::ErrorKind::InvalidData && e.to_string().contains("Invalid header") => {
        re_fetch_or_regenerate(&path)? // empty/truncated/wrong file
    }
    other => other?,
}

Prevention

When it happens

Trigger: UniversalHashMap::open on an empty (0-byte) file, a file smaller than the header, a placeholder file created ahead of the real data, or a completely wrong file at the expected path.

Common situations: A download/replication step failed silently and left an empty file; a pre-created placeholder was never replaced; pointing the open call at the wrong path (e.g., a bitmask or tar file instead of the map).

Related errors

AI-assisted analysis of qdrant/qdrant@74f3e85b94 (2026-08-22). Data as JSON: /api/errors/dbd7a5dc8ccc8e51. Report an issue: GitHub.