What causes "File too small to be a valid gzip file" in nautechsystems/nautilus_trader?

Re-download or regenerate the dataset file; verify it has non-trivial size (ls -l)

File too small to be a valid gzip file

Error message

File too small to be a valid gzip file

What it means

create_csv_reader inspects the first two bytes to decide whether the file is gzipped (gzip magic 0x1f 0x8b). Files smaller than 2 bytes cannot possibly contain a valid gzip header, so it bails with this message before attempting to read the magic bytes.

Source

Thrown at crates/adapters/tardis/src/csv/mod.rs:122

    let filepath_ref = filepath.as_ref();
    let mut file = open_file_with_retry(filepath_ref, MAX_RETRIES, DELAY_MS)?;

    let is_gzipped = filepath_ref
        .extension()
        .and_then(OsStr::to_str)
        .is_some_and(|ext| ext.eq_ignore_ascii_case("gz"));

    if !is_gzipped {
        let buf_reader = BufReader::with_capacity(BUFFER_SIZE, file);
        return Ok(ReaderBuilder::new()
            .has_headers(true)
            .buffer_capacity(1024 * 1024) // 1MB CSV buffer
            .from_reader(Box::new(buf_reader)));
    }

    let file_size = file.metadata()?.len();
    if file_size < 2 {
        anyhow::bail!("File too small to be a valid gzip file");
    }

    let mut header_buf = [0u8; 2];
    for attempt in 1..=MAX_RETRIES {
        match file.read_exact(&mut header_buf) {
            Ok(()) => break,
            Err(e) => {
                if attempt == MAX_RETRIES {
                    anyhow::bail!(
                        "Failed to read gzip header from '{}' after {MAX_RETRIES} attempts: {e}",
                        filepath_ref.display()
                    );
                }
                log::warn!(
                    "Attempt {attempt}/{MAX_RETRIES} failed to read header from '{}': {e}. Retrying after {DELAY_MS}ms...",
                    filepath_ref.display()
                );
                std::thread::sleep(Duration::from_millis(DELAY_MS));

View on GitHub (pinned to 18893faf8b)

Solutions

Re-download or regenerate the dataset file; verify it has non-trivial size (ls -l)
Validate downloads by checking file size and gzip integrity (gzip -t) before loading
Fix the upstream pipeline step that produced the empty file
Point the loader at the correct file path — you may be reading a placeholder at the wrong location

Defensive patterns

Strategy: validation

Validate before calling

fn is_plausible_gzip(path: &Path) -> std::io::Result<bool> {
    let len = std::fs::metadata(path)?.len();
    Ok(len >= 2)
}
// call before create_csv_reader; also run `gzip -t` for full integrity

Try / catch

match create_csv_reader(path) {
    Err(e) if e.to_string().contains("File too small to be a valid gzip file") => {
        log::error!("dataset file is empty/truncated: {e}");
        // re-download the dataset
    }
    Ok(r) => /* proceed */,
    Err(e) => return Err(e),
}

Prevention

Check file size > 0 (ideally > 2 bytes) before loading
Validate downloads with gzip -t or checksum after transfer
Fix pipeline steps that can emit empty placeholder files

When it happens

Trigger: Calling create_csv_reader (directly or via load_trades/load_quotes/load_deltas/load_depth10_*/convert_options_chain_csv) with a file whose size is 0 or 1 bytes — an empty or truncated download.

Common situations: Tardis dataset export interrupted leaving a 0-byte .csv.gz; failed download producing an empty placeholder file; touching a file that was never populated.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/bbc88bb989da1145. Report an issue: GitHub.