nautechsystems/nautilus_trader · error · anyhow::Error

Failed to create StreamReader: {e}

Error message

Failed to create StreamReader: {e}

What it means

Raised in `read_feather_file` when `StreamReader::try_new` from the Arrow IPC crate cannot open a feather (Arrow IPC stream) file fetched from the object store. This happens during initial header/schema parsing, meaning the bytes are not a valid Arrow IPC stream before any batches are read.

Source

Thrown at crates/persistence/src/backend/catalog.rs:3819

    /// and returns all `RecordBatches` contained within it.
    fn read_feather_file(&self, file_path: &str) -> anyhow::Result<Vec<RecordBatch>> {
        use datafusion::arrow::ipc::reader::StreamReader;

        let bytes = self.execute_async(async {
            let path = ObjectPath::from(file_path);
            let result = self.object_store.get(&path).await?;
            let bytes = result.bytes().await?;
            Ok::<_, anyhow::Error>(bytes)
        })?;

        if bytes.is_empty() {
            return Ok(Vec::new());
        }

        // Read the Arrow IPC stream
        let cursor = Cursor::new(bytes.as_ref());
        let reader = StreamReader::try_new(cursor, None)
            .map_err(|e| anyhow::anyhow!("Failed to create StreamReader: {e}"))?;

        // Read all batches
        let mut batches = Vec::new();

        for batch_result in reader {
            let batch = batch_result.map_err(|e| anyhow::anyhow!("Failed to read batch: {e}"))?;
            batches.push(batch);
        }

        Ok(batches)
    }

    /// Converts `RecordBatches` to Data objects, optionally replacing `ts_init` with `ts_event`.
    fn convert_record_batches_to_data<T>(
        batches: Vec<RecordBatch>,
        use_ts_event_for_ts_init: bool,
    ) -> anyhow::Result<Vec<T>>
    where

View on GitHub (pinned to 18893faf8b)

Solutions

1. Open the file with pyarrow (`pa.ipc.open_stream`) or arrow-rs to confirm it is a valid Arrow IPC stream; regenerate the file if not.
2. Verify the file was written in stream (not file/table) IPC format; re-export with the current writer if it is a legacy file-format file.
3. Check file size and re-download/re-write — truncated or partially-uploaded files fail header parsing.
4. Confirm the object store path in the error points at the intended file, not a misnamed or placeholder file.
5. If upgrading from an older release, re-run the stream→catalog conversion from the original raw data.

// before conversion, sanity-check the file
let bytes = fetch(file_path)?;
if bytes.len() < 8 || &bytes[..4] != b"ARRO".map(|b: u8| b) {} // prefer: try opening with pa.ipc.open_stream in a check tool

match catalog.read_feather_file_result(path) {
    Ok(batches) => batches,
    Err(e) if e.to_string().contains("Failed to create StreamReader") => {
        // quarantine/rewrite the invalid feather file, then continue
        quarantine(path); Vec::new()
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling `read_run_data` or `convert_stream_to_data` where the discovered `.feather` file's bytes fail IPC stream validation: file truncated/empty-but-nonzero, wrong format (e.g. a Parquet file named `.feather`, or an Arrow File-format file rather than stream format), or corrupted download from the object store.

Common situations: Stream files written by an older Nautilus version with a different IPC format; files renamed or converted incorrectly; interrupted uploads leaving partial bytes; pointing the catalog at a directory where the `.feather` extension was applied to non-feather data.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to read batch: {e}
• height must be positive, was {self.height}
• Cannot convert empty stream batch to parquet
• custom data type '{type_name}' is not registered with an Arr
• e.to_string()

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/ac08164d4c8c7244. Report an issue: GitHub.