nautechsystems/nautilus_trader · error

Failed to read gzip header from '{}' after {MAX_RETRIES} att

Error message

Failed to read gzip header from '{}' after {MAX_RETRIES} attempts: {e}

What it means

create_csv_reader reads the first two bytes of a non-trivial file to detect gzip magic. If read_exact fails on every one of MAX_RETRIES attempts, it bails with this message including the file path, attempt count, and the last IO error.

Source

Thrown at crates/adapters/tardis/src/csv/mod.rs:131

        let buf_reader = BufReader::with_capacity(BUFFER_SIZE, file);
        return Ok(ReaderBuilder::new()
            .has_headers(true)
            .buffer_capacity(1024 * 1024) // 1MB CSV buffer
            .from_reader(Box::new(buf_reader)));
    }

    let file_size = file.metadata()?.len();
    if file_size < 2 {
        anyhow::bail!("File too small to be a valid gzip file");
    }

    let mut header_buf = [0u8; 2];
    for attempt in 1..=MAX_RETRIES {
        match file.read_exact(&mut header_buf) {
            Ok(()) => break,
            Err(e) => {
                if attempt == MAX_RETRIES {
                    anyhow::bail!(
                        "Failed to read gzip header from '{}' after {MAX_RETRIES} attempts: {e}",
                        filepath_ref.display()
                    );
                }
                log::warn!(
                    "Attempt {attempt}/{MAX_RETRIES} failed to read header from '{}': {e}. Retrying after {DELAY_MS}ms...",
                    filepath_ref.display()
                );
                std::thread::sleep(Duration::from_millis(DELAY_MS));
            }
        }
    }

    if header_buf[0] != 0x1f || header_buf[1] != 0x8b {
        anyhow::bail!(
            "File '{}' has .gz extension but invalid gzip header",
            filepath_ref.display()
        );

View on GitHub (pinned to 18893faf8b)

Solutions

1. Check file readability/permissions for the process running the loader
2. Inspect the underlying error {e}: UnexpectedEof implies the file was truncated concurrently — regenerate it; others indicate IO/permission issues
3. Ensure no other process is writing to the file while it is loaded; load only finalized files
4. Retry the load; if on a network filesystem, verify mount stability

fn is_readable(path: &Path) -> bool {
    use std::io::Read;
    std::fs::File::open(path)
        .and_then(|mut f| {
            let mut b = [0u8; 2];
            f.read_exact(&mut b)
        })
        .is_ok()
}

match create_csv_reader(path) {
    Err(e) if e.to_string().contains("Failed to read gzip header") => {
        log::error!("cannot read file header: {e:#}");
        // check permissions/concurrent writers, then retry
    }
    Ok(r) => /* proceed */,
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling create_csv_reader on an existing, non-trivially-sized file whose first 2 bytes cannot be read after retries — typically an unreadable file (permissions), an IO error on the storage layer, or a race where the file shrinks/vanishes between metadata check and read.

Common situations: Permission problems on dataset files; network mounts dropping mid-read; concurrent writers truncating the file while it is being inspected; disk/IO hardware errors.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to open file '{}' after {max_retries} attempts: {e}
• File too small to be a valid gzip file
• File '{}' has .gz extension but invalid gzip header
• options_chain instrument derivation supports Deribit only, r
• CSV file is empty

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/1078427c4f7858b4. Report an issue: GitHub.