{"record":{"id":"7e73bc424471679e","repo":"nautechsystems/nautilus_trader","slug":"failed-to-read-batch-e","errorCode":null,"errorMessage":"Failed to read batch: {e}","messagePattern":"Failed to read batch: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog.rs","lineNumber":3825,"sourceCode":"            let result = self.object_store.get(&path).await?;\n            let bytes = result.bytes().await?;\n            Ok::<_, anyhow::Error>(bytes)\n        })?;\n\n        if bytes.is_empty() {\n            return Ok(Vec::new());\n        }\n\n        // Read the Arrow IPC stream\n        let cursor = Cursor::new(bytes.as_ref());\n        let reader = StreamReader::try_new(cursor, None)\n            .map_err(|e| anyhow::anyhow!(\"Failed to create StreamReader: {e}\"))?;\n\n        // Read all batches\n        let mut batches = Vec::new();\n\n        for batch_result in reader {\n            let batch = batch_result.map_err(|e| anyhow::anyhow!(\"Failed to read batch: {e}\"))?;\n            batches.push(batch);\n        }\n\n        Ok(batches)\n    }\n\n    /// Converts `RecordBatches` to Data objects, optionally replacing `ts_init` with `ts_event`.\n    fn convert_record_batches_to_data<T>(\n        batches: Vec<RecordBatch>,\n        use_ts_event_for_ts_init: bool,\n    ) -> anyhow::Result<Vec<T>>\n    where\n        T: DecodeDataFromRecordBatch + TryFrom<Data>,\n    {\n        Self::convert_record_batches_to_data_with_bar_type_conversion(\n            batches,\n            use_ts_event_for_ts_init,\n            false,","sourceCodeStart":3807,"sourceCodeEnd":3843,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog.rs#L3807-L3843","documentation":"Raised in `read_feather_file` while iterating the Arrow `StreamReader`: the stream header parsed successfully, but a subsequent record batch failed to decode. The reader surfaces the Arrow error (schema mismatch, corrupt data section, unexpected EOF) wrapped in this message.","triggerScenarios":"Reading a `.feather` file via `read_run_data`/`convert_stream_to_data` where a mid-file record batch is corrupt or truncated — e.g. bytes end abruptly inside a batch, a batch's buffers disagree with the declared schema, or the file mixes incompatible IPC versions.","commonSituations":"Partially uploaded/synced files (first batches fine, tail truncated); disk or network corruption in object storage; files produced by a buggy or incompatible writer version; concurrent writes while the catalog reads the file.","solutions":["Re-write or re-upload the feather file — truncation mid-batch is the most common cause; compare file size against the source.","Determine how many batches read successfully before the failure to locate the corruption point (the batch index in the iteration).","Regenerate the stream data from the original run rather than trying to salvage the damaged file.","Check the writer version: if the file came from an older NautilusTrader, re-export with the current version.","Avoid reading files while a writer may still be appending to the same object-store key."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// compare remote size vs source size before reading\nassert_eq!(object_store_size(path)?, local_source_size(path)?, \"truncated file\");","typeGuard":null,"tryCatchPattern":"for attempt in 0..3 {\n    match read_and_decode(path) {\n        Ok(v) => break v,\n        Err(e) if e.to_string().contains(\"Failed to read batch\") && attempt < 2 => continue, // re-fetch bytes\n        Err(e) => return Err(e),\n    }\n}","preventionTips":["Ensure uploads/syncs are atomic so readers never see partially written files.","Checksum feather files after transfer and before conversion.","Don't write to the same object-store key while a conversion job is reading it.","Regenerate from raw data rather than salvaging a file that fails mid-batch."],"tags":["arrow","ipc","feather","corruption"],"backgroundTag":"file-read-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}