quickwit-oss/tantivy · error · io::Error

UnexpectedEof

UnexpectedEof

Error message

File corrupted. The file is smaller than 4 bytes (len={}).

What it means

Every tantivy segment file ends with a footer whose minimum size is 4 bytes (the footer magic number plus metadata). extract_footer rejects any file smaller than 4 bytes because it cannot possibly contain a valid footer, so the file is treated as corrupted from the start.

Source

Thrown at src/directory/footer.rs:56

        Footer { version, crc }
    }

    pub(crate) fn crc(&self) -> CrcHashU32 {
        self.crc
    }
    pub(crate) fn append_footer<W: io::Write>(&self, mut write: &mut W) -> io::Result<()> {
        let mut counting_write = CountingWriter::wrap(&mut write);
        counting_write.write_all(serde_json::to_string(&self)?.as_ref())?;
        let footer_payload_len = counting_write.written_bytes();
        BinarySerializable::serialize(&(footer_payload_len as u32), write)?;
        BinarySerializable::serialize(&FOOTER_MAGIC_NUMBER, write)?;
        Ok(())
    }

    /// Extracts the tantivy Footer from the file and returns the footer and the rest of the file
    pub fn extract_footer(file: FileSlice) -> io::Result<(Footer, FileSlice)> {
        if file.len() < 4 {
            return Err(io::Error::new(
                io::ErrorKind::UnexpectedEof,
                format!(
                    "File corrupted. The file is smaller than 4 bytes (len={}).",
                    file.len()
                ),
            ));
        }

        let footer_metadata_len = <(u32, u32)>::SIZE_IN_BYTES;
        let (footer_len, footer_magic_byte): (u32, u32) = file
            .slice_from_end(footer_metadata_len)
            .read_bytes()?
            .as_ref()
            .deserialize()?;

        if footer_magic_byte != FOOTER_MAGIC_NUMBER {
            return Err(io::Error::new(
                io::ErrorKind::InvalidData,

View on GitHub (pinned to b5d8deb80c)

Solutions

  1. Delete or restore the undersized file; it cannot contain valid data
  2. Rebuild the index or restore it from a backup
  3. Check for disk-full or interrupted-write issues on the source filesystem
  4. Compare the directory against commit metadata (meta.json) to identify missing/corrupt segments
Defensive patterns

Strategy: validation

Validate before calling

let md = std::fs::metadata(path)?;
if md.len() < 4 { /* skip / quarantine / restore this file */ }

Try / catch

match extract_footer(file_slice) {
    Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => {
        eprintln!("file too small for footer, restoring from backup");
    }
    Err(e) => return Err(e),
    Ok((footer, rest)) => (footer, rest),
}

Prevention

When it happens

Trigger: Calling extract_footer (or opening an index) on a zero-length or 1-3 byte file: empty file created by a failed write, placeholder/lock files misread as segments, truncated uploads.

Common situations: Disk-full or crash during index write leaving empty files; incomplete rsync/scp of the index directory; pointing tantivy at the wrong directory containing stray empty files.

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/75348127f8438b91. Report an issue: GitHub.