nautechsystems/nautilus_trader · error · anyhow::Error

Failed to read batch: {e}

Error message

Failed to read batch: {e}

What it means

Raised in `read_feather_file` while iterating the Arrow `StreamReader`: the stream header parsed successfully, but a subsequent record batch failed to decode. The reader surfaces the Arrow error (schema mismatch, corrupt data section, unexpected EOF) wrapped in this message.

Source

Thrown at crates/persistence/src/backend/catalog.rs:3825

            let result = self.object_store.get(&path).await?;
            let bytes = result.bytes().await?;
            Ok::<_, anyhow::Error>(bytes)
        })?;

        if bytes.is_empty() {
            return Ok(Vec::new());
        }

        // Read the Arrow IPC stream
        let cursor = Cursor::new(bytes.as_ref());
        let reader = StreamReader::try_new(cursor, None)
            .map_err(|e| anyhow::anyhow!("Failed to create StreamReader: {e}"))?;

        // Read all batches
        let mut batches = Vec::new();

        for batch_result in reader {
            let batch = batch_result.map_err(|e| anyhow::anyhow!("Failed to read batch: {e}"))?;
            batches.push(batch);
        }

        Ok(batches)
    }

    /// Converts `RecordBatches` to Data objects, optionally replacing `ts_init` with `ts_event`.
    fn convert_record_batches_to_data<T>(
        batches: Vec<RecordBatch>,
        use_ts_event_for_ts_init: bool,
    ) -> anyhow::Result<Vec<T>>
    where
        T: DecodeDataFromRecordBatch + TryFrom<Data>,
    {
        Self::convert_record_batches_to_data_with_bar_type_conversion(
            batches,
            use_ts_event_for_ts_init,
            false,

View on GitHub (pinned to 18893faf8b)

Solutions

1. Re-write or re-upload the feather file — truncation mid-batch is the most common cause; compare file size against the source.
2. Determine how many batches read successfully before the failure to locate the corruption point (the batch index in the iteration).
3. Regenerate the stream data from the original run rather than trying to salvage the damaged file.
4. Check the writer version: if the file came from an older NautilusTrader, re-export with the current version.
5. Avoid reading files while a writer may still be appending to the same object-store key.

// compare remote size vs source size before reading
assert_eq!(object_store_size(path)?, local_source_size(path)?, "truncated file");

for attempt in 0..3 {
    match read_and_decode(path) {
        Ok(v) => break v,
        Err(e) if e.to_string().contains("Failed to read batch") && attempt < 2 => continue, // re-fetch bytes
        Err(e) => return Err(e),
    }
}

When it happens

Trigger: Reading a `.feather` file via `read_run_data`/`convert_stream_to_data` where a mid-file record batch is corrupt or truncated — e.g. bytes end abruptly inside a batch, a batch's buffers disagree with the declared schema, or the file mixes incompatible IPC versions.

Common situations: Partially uploaded/synced files (first batches fine, tail truncated); disk or network corruption in object storage; files produced by a buggy or incompatible writer version; concurrent writes while the catalog reads the file.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to create StreamReader: {e}
• Failed to apply stream conversion transforms for {feather_pa
• height must be positive, was {self.height}
• pandas is required for report generation; install it with `p
• pandas is required for visualization; install it with `pip i

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/7e73bc424471679e. Report an issue: GitHub.