quickwit-oss/tantivy · error · io::Error

File corrupted. The file is smaller than Footer::SIZE_IN_BYT

Error message

File corrupted. The file is smaller than Footer::SIZE_IN_BYTES (len={}).

What it means

Thrown by DocStoreFooter::extract_footer when the file slice is shorter than the fixed footer size (DocStoreFooter::SIZE_IN_BYTES). A doc store file must end with a footer; a file too small to contain one cannot be a valid doc store file.

Source

Thrown at src/store/footer.rs:66

    const SIZE_IN_BYTES: usize = 28;
}

impl DocStoreFooter {
    pub fn new(
        offset: u64,
        decompressor: Decompressor,
        doc_store_version: DocStoreVersion,
    ) -> Self {
        DocStoreFooter {
            offset,
            doc_store_version,
            decompressor,
        }
    }

    pub fn extract_footer(file: FileSlice) -> io::Result<(DocStoreFooter, FileSlice)> {
        if file.len() < DocStoreFooter::SIZE_IN_BYTES {
            return Err(io::Error::new(
                io::ErrorKind::UnexpectedEof,
                format!(
                    "File corrupted. The file is smaller than Footer::SIZE_IN_BYTES (len={}).",
                    file.len()
                ),
            ));
        }
        let (body, footer_slice) = file.split_from_end(DocStoreFooter::SIZE_IN_BYTES);
        let mut footer_bytes = footer_slice.read_bytes()?;
        let footer = DocStoreFooter::deserialize(&mut footer_bytes)?;
        Ok((footer, body))
    }
}

#[test]
fn doc_store_footer_test() {
    // This test is just to safe guard changes on the footer.
    // When the doc store footer is updated, make sure to update also the serialize/deserialize

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Verify the path points to a real, complete doc store/segment file and not an empty or wrong file
2. Re-copy the index directory fully and compare file sizes against the source
3. Restore the index from backup or re-index the documents
4. Check for external processes (cleanup jobs, disk-full events) that may have truncated files

Example fix

// before
let (footer, rest) = DocStoreFooter::extract_footer(file_slice)?;
// after
if file_slice.len() < DocStoreFooter::SIZE_IN_BYTES {
    eprintln!("skipping invalid/truncated doc store file of {} bytes", file_slice.len());
    return Ok(None);
}
let (footer, rest) = Some(DocStoreFooter::extract_footer(file_slice)?).map(|(f, r)| (f, r))?;

use std::io;
use owning_ref::FileSlice; // or your FileSlice type

fn footer_is_readable(file: &FileSlice) -> bool {
    file.len() >= DocStoreFooter::SIZE_IN_BYTES
}

// call before extract_footer:
if !footer_is_readable(&file_slice) {
    eprintln!("file of {} bytes is too small to hold a doc store footer; skipping", file_slice.len());
}

fn is_valid_docstore_file(file: &FileSlice) -> bool {
    file.len() >= DocStoreFooter::SIZE_IN_BYTES
}

match DocStoreFooter::extract_footer(file_slice) {
    Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => {
        eprintln!("file too small for footer; treating as invalid doc store");
        // skip or re-index
    }
    other => other.map(|(footer, rest)| footer),
}

When it happens

Trigger: Calling extract_footer() on an empty or nearly empty file; passing the wrong file (not a doc store segment) to the footer parser; reading a file truncated to fewer bytes than the footer requires.

Common situations: Pointing tantivy at an empty or wrong path; incomplete download/copy of index directories; files clobbered or truncated by external tooling.

Related errors

• doc store block not completely decompressed, data corruption
• failed to read block_len
• failed to read block content
• InvalidData
• UnexpectedEof

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/b3e1d33e2b416762. Report an issue: GitHub.