{"record":{"id":"bbc88bb989da1145","repo":"nautechsystems/nautilus_trader","slug":"file-too-small-to-be-a-valid-gzip-file","errorCode":null,"errorMessage":"File too small to be a valid gzip file","messagePattern":"File too small to be a valid gzip file","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/adapters/tardis/src/csv/mod.rs","lineNumber":122,"sourceCode":"    let filepath_ref = filepath.as_ref();\n    let mut file = open_file_with_retry(filepath_ref, MAX_RETRIES, DELAY_MS)?;\n\n    let is_gzipped = filepath_ref\n        .extension()\n        .and_then(OsStr::to_str)\n        .is_some_and(|ext| ext.eq_ignore_ascii_case(\"gz\"));\n\n    if !is_gzipped {\n        let buf_reader = BufReader::with_capacity(BUFFER_SIZE, file);\n        return Ok(ReaderBuilder::new()\n            .has_headers(true)\n            .buffer_capacity(1024 * 1024) // 1MB CSV buffer\n            .from_reader(Box::new(buf_reader)));\n    }\n\n    let file_size = file.metadata()?.len();\n    if file_size < 2 {\n        anyhow::bail!(\"File too small to be a valid gzip file\");\n    }\n\n    let mut header_buf = [0u8; 2];\n    for attempt in 1..=MAX_RETRIES {\n        match file.read_exact(&mut header_buf) {\n            Ok(()) => break,\n            Err(e) => {\n                if attempt == MAX_RETRIES {\n                    anyhow::bail!(\n                        \"Failed to read gzip header from '{}' after {MAX_RETRIES} attempts: {e}\",\n                        filepath_ref.display()\n                    );\n                }\n                log::warn!(\n                    \"Attempt {attempt}/{MAX_RETRIES} failed to read header from '{}': {e}. Retrying after {DELAY_MS}ms...\",\n                    filepath_ref.display()\n                );\n                std::thread::sleep(Duration::from_millis(DELAY_MS));","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/tardis/src/csv/mod.rs#L104-L140","documentation":"create_csv_reader inspects the first two bytes to decide whether the file is gzipped (gzip magic 0x1f 0x8b). Files smaller than 2 bytes cannot possibly contain a valid gzip header, so it bails with this message before attempting to read the magic bytes.","triggerScenarios":"Calling create_csv_reader (directly or via load_trades/load_quotes/load_deltas/load_depth10_*/convert_options_chain_csv) with a file whose size is 0 or 1 bytes — an empty or truncated download.","commonSituations":"Tardis dataset export interrupted leaving a 0-byte .csv.gz; failed download producing an empty placeholder file; touching a file that was never populated.","solutions":["Re-download or regenerate the dataset file; verify it has non-trivial size (ls -l)","Validate downloads by checking file size and gzip integrity (gzip -t) before loading","Fix the upstream pipeline step that produced the empty file","Point the loader at the correct file path — you may be reading a placeholder at the wrong location"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"fn is_plausible_gzip(path: &Path) -> std::io::Result<bool> {\n    let len = std::fs::metadata(path)?.len();\n    Ok(len >= 2)\n}\n// call before create_csv_reader; also run `gzip -t` for full integrity","typeGuard":null,"tryCatchPattern":"match create_csv_reader(path) {\n    Err(e) if e.to_string().contains(\"File too small to be a valid gzip file\") => {\n        log::error!(\"dataset file is empty/truncated: {e}\");\n        // re-download the dataset\n    }\n    Ok(r) => /* proceed */,\n    Err(e) => return Err(e),\n}","preventionTips":["Check file size > 0 (ideally > 2 bytes) before loading","Validate downloads with gzip -t or checksum after transfer","Fix pipeline steps that can emit empty placeholder files"],"tags":["tardis","csv","gzip","file-validation"],"backgroundTag":"invalid-file-content","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}