{"record":{"id":"1078427c4f7858b4","repo":"nautechsystems/nautilus_trader","slug":"failed-to-read-gzip-header-from-after-max-re","errorCode":null,"errorMessage":"Failed to read gzip header from '{}' after {MAX_RETRIES} attempts: {e}","messagePattern":"Failed to read gzip header from '(.+?)' after (.+?) attempts: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/adapters/tardis/src/csv/mod.rs","lineNumber":131,"sourceCode":"        let buf_reader = BufReader::with_capacity(BUFFER_SIZE, file);\n        return Ok(ReaderBuilder::new()\n            .has_headers(true)\n            .buffer_capacity(1024 * 1024) // 1MB CSV buffer\n            .from_reader(Box::new(buf_reader)));\n    }\n\n    let file_size = file.metadata()?.len();\n    if file_size < 2 {\n        anyhow::bail!(\"File too small to be a valid gzip file\");\n    }\n\n    let mut header_buf = [0u8; 2];\n    for attempt in 1..=MAX_RETRIES {\n        match file.read_exact(&mut header_buf) {\n            Ok(()) => break,\n            Err(e) => {\n                if attempt == MAX_RETRIES {\n                    anyhow::bail!(\n                        \"Failed to read gzip header from '{}' after {MAX_RETRIES} attempts: {e}\",\n                        filepath_ref.display()\n                    );\n                }\n                log::warn!(\n                    \"Attempt {attempt}/{MAX_RETRIES} failed to read header from '{}': {e}. Retrying after {DELAY_MS}ms...\",\n                    filepath_ref.display()\n                );\n                std::thread::sleep(Duration::from_millis(DELAY_MS));\n            }\n        }\n    }\n\n    if header_buf[0] != 0x1f || header_buf[1] != 0x8b {\n        anyhow::bail!(\n            \"File '{}' has .gz extension but invalid gzip header\",\n            filepath_ref.display()\n        );","sourceCodeStart":113,"sourceCodeEnd":149,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/tardis/src/csv/mod.rs#L113-L149","documentation":"create_csv_reader reads the first two bytes of a non-trivial file to detect gzip magic. If read_exact fails on every one of MAX_RETRIES attempts, it bails with this message including the file path, attempt count, and the last IO error.","triggerScenarios":"Calling create_csv_reader on an existing, non-trivially-sized file whose first 2 bytes cannot be read after retries — typically an unreadable file (permissions), an IO error on the storage layer, or a race where the file shrinks/vanishes between metadata check and read.","commonSituations":"Permission problems on dataset files; network mounts dropping mid-read; concurrent writers truncating the file while it is being inspected; disk/IO hardware errors.","solutions":["Check file readability/permissions for the process running the loader","Inspect the underlying error {e}: UnexpectedEof implies the file was truncated concurrently — regenerate it; others indicate IO/permission issues","Ensure no other process is writing to the file while it is loaded; load only finalized files","Retry the load; if on a network filesystem, verify mount stability"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"fn is_readable(path: &Path) -> bool {\n    use std::io::Read;\n    std::fs::File::open(path)\n        .and_then(|mut f| {\n            let mut b = [0u8; 2];\n            f.read_exact(&mut b)\n        })\n        .is_ok()\n}","typeGuard":null,"tryCatchPattern":"match create_csv_reader(path) {\n    Err(e) if e.to_string().contains(\"Failed to read gzip header\") => {\n        log::error!(\"cannot read file header: {e:#}\");\n        // check permissions/concurrent writers, then retry\n    }\n    Ok(r) => /* proceed */,\n    Err(e) => return Err(e),\n}","preventionTips":["Check file permissions for the loading process","Never read files still being written by another process","Retry loads on network filesystems and verify mount health","Investigate the wrapped IO error to distinguish truncation from permission issues"],"tags":["tardis","csv","gzip","file-io"],"backgroundTag":"file-read-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}