nautechsystems/nautilus_trader · error

File '{}' has .gz extension but invalid gzip header

Error message

File '{}' has .gz extension but invalid gzip header

What it means

create_csv_reader validates that files with a .gz extension actually begin with the gzip magic bytes (0x1f 0x8b) before handing the reader to CSV parsing. The library throws this when the extension claims gzip compression but the file content is not a valid gzip stream, which would otherwise cause cryptic decompression failures downstream.

Source

Thrown at crates/adapters/tardis/src/csv/mod.rs:146

            Ok(()) => break,
            Err(e) => {
                if attempt == MAX_RETRIES {
                    anyhow::bail!(
                        "Failed to read gzip header from '{}' after {MAX_RETRIES} attempts: {e}",
                        filepath_ref.display()
                    );
                }
                log::warn!(
                    "Attempt {attempt}/{MAX_RETRIES} failed to read header from '{}': {e}. Retrying after {DELAY_MS}ms...",
                    filepath_ref.display()
                );
                std::thread::sleep(Duration::from_millis(DELAY_MS));
            }
        }
    }

    if header_buf[0] != 0x1f || header_buf[1] != 0x8b {
        anyhow::bail!(
            "File '{}' has .gz extension but invalid gzip header",
            filepath_ref.display()
        );
    }

    for attempt in 1..=MAX_RETRIES {
        match file.seek(SeekFrom::Start(0)) {
            Ok(_) => break,
            Err(e) => {
                if attempt == MAX_RETRIES {
                    anyhow::bail!(
                        "Failed to reset file position for '{}' after {MAX_RETRIES} attempts: {e}",
                        filepath_ref.display()
                    );
                }
                log::warn!(
                    "Attempt {attempt}/{MAX_RETRIES} failed to seek in '{}': {e}. Retrying after {DELAY_MS}ms...",
                    filepath_ref.display()

View on GitHub (pinned to 18893faf8b)

Solutions

1. Verify the file is genuinely gzipped: run `file data.csv.gz` or `gunzip -t data.csv.gz`; re-download from Tardis if corrupt
2. If the file is actually uncompressed, rename it to .csv (remove the .gz extension)
3. Re-download the archive with a proper HTTP client, checking the response is a gzip stream
4. Confirm no proxy/CDN stripped the Content-Encoding during download

Example fix

// before (misnamed uncompressed file)
loader.load_trades("trades.csv.gz")?;
// after
gunzip -t trades.csv.gz  # verify first; if uncompressed:
// loader.load_trades("trades.csv")?;  (or `mv trades.csv.gz trades.csv`)

fn is_gzip(path: &Path) -> anyhow::Result<bool> {
    use std::io::Read;
    let mut f = std::fs::File::open(path)?;
    let mut b = [0u8; 2];
    f.read_exact(&mut b)?;
    Ok(b == [0x1f, 0x8b])
}
// before loading:
assert!(is_gzip(&path)?, "{} is not a valid gzip file", path.display());

match loader.load_trades(&path) {
    Ok(data) => data,
    Err(e) if e.to_string().contains("invalid gzip header") => {
        eprintln!("File is not gzip; re-download or rename to .csv");
        return Err(e);
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling any Tardis CSV loader (load_trades, load_quotes, load_deltas, load_depth10_from_snapshot5, load_depth10_from_snapshot25, convert_options_chain_csv) with a filepath ending in .gz whose first two bytes are not 0x1f 0x8b — e.g. an uncompressed file renamed to .gz, or a truncated/HTML error page saved as .gz.

Common situations: A Tardis download failed and returned an HTML/JSON error page saved with the requested .gz name; a file was downloaded over plain HTTP without compression; a user gunzipped a file but kept the .gz extension.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• File too small to be a valid gzip file
• Failed to read gzip header from '{}' after {MAX_RETRIES} att
• options_chain instrument derivation supports Deribit only, r
• Failed to open file '{}' after {max_retries} attempts: {e}
• CSV file is empty

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/133bc9ecbe309b39. Report an issue: GitHub.