nautechsystems/nautilus_trader · error

Failed to convert batched deltas to Python: {e}

Error message

Failed to convert batched deltas to Python: {e}

What it means

This Python-facing stream iterator converts each completed batch of `OrderBookDelta`s into Python objects via `data_to_pyobject` inside `Python::with_gil`. If that conversion returns a `PyResult` error (e.g. a type-conversion or interpreter failure), it is mapped into an anyhow error with this message before being yielded from `next`.

Source

Thrown at crates/adapters/tardis/src/csv/stream.rs:583

        if let Some(Err(e)) = self.fill_pending_batches() {
            return Some(Err(e));
        }

        if self.pending_batches.is_empty() {
            None
        } else {
            let batches = std::mem::take(&mut self.pending_batches);
            let result = Python::attach(|py| {
                batches
                    .into_iter()
                    .map(|batch| {
                        let deltas = OrderBookDeltas::new(self.instrument_id, batch);
                        let deltas = Box::new(deltas);
                        data_to_pyobject(py, Data::BookDeltas(deltas))
                    })
                    .collect::<PyResult<Vec<_>>>()
            })
            .map_err(|e| anyhow::anyhow!("Failed to convert batched deltas to Python: {e}"));
            Some(result)
        }
    }
}

#[cfg(feature = "python")]
/// Streams batches of `OrderBookDeltas` Python objects from a Tardis format CSV at the given
/// `filepath`, yielding chunks of the specified size.
///
/// # Errors
///
/// Returns an error if `chunk_size` is outside `[1, 1_000_000]`, or if the file cannot be opened,
/// read, or parsed as CSV.
pub fn stream_batched_deltas<P: AsRef<Path>>(
    filepath: P,
    chunk_size: usize,
    price_precision: Option<u8>,
    size_precision: Option<u8>,

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Ensure the stream is fully consumed (or dropped) before the Python interpreter shuts down; close iterators explicitly in `finally` blocks.
  2. Inspect the inner `{e}` (PyErr) message for the root cause — it names the actual conversion failure.
  3. Avoid holding stream iterators across interpreter boundaries or threads without the GIL context.
  4. Update nautilus_trader if the failure stems from a known pyo3 conversion bug in your installed version.

Example fix

// before
for deltas in stream:  # interpreter may finalize mid-loop
    process(deltas)
# after
try:
    for deltas in stream:
        process(deltas)
finally:
    stream.close()  # release the Rust iterator before shutdown
Defensive patterns

Strategy: try-catch

Try / catch

stream = delta_stream(path, instrument_id, chunk_size)
try:
    while True:
        batch = stream.next()
        if batch is None:
            break
        process(batch)
except Exception as e:
    if "Failed to convert batched deltas to Python" in str(e):
        log.error("pyo3 conversion failed (interpreter state?): %s", e)
        raise
    raise
finally:
    stream.close()  # release the Rust iterator before interpreter shutdown

Prevention

When it happens

Trigger: Calling the Python `next()` on the delta stream when the Rust-to-Python conversion of a batched `OrderBookDeltas` fails — typically an error raised in Python object construction under the GIL, or a returned `PyErr` from the conversion helper.

Common situations: Python interpreter in a bad state (interpreter finalization during shutdown while the stream is still alive); a failing conversion for a particular instrument_id/delta payload; mixing stream objects across interpreter contexts or after `sys.exit` in scripts/notebooks.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/587106beb4edf85b. Report an issue: GitHub.