{"record":{"id":"587106beb4edf85b","repo":"nautechsystems/nautilus_trader","slug":"failed-to-convert-batched-deltas-to-python-e","errorCode":null,"errorMessage":"Failed to convert batched deltas to Python: {e}","messagePattern":"Failed to convert batched deltas to Python: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/adapters/tardis/src/csv/stream.rs","lineNumber":583,"sourceCode":"        if let Some(Err(e)) = self.fill_pending_batches() {\n            return Some(Err(e));\n        }\n\n        if self.pending_batches.is_empty() {\n            None\n        } else {\n            let batches = std::mem::take(&mut self.pending_batches);\n            let result = Python::attach(|py| {\n                batches\n                    .into_iter()\n                    .map(|batch| {\n                        let deltas = OrderBookDeltas::new(self.instrument_id, batch);\n                        let deltas = Box::new(deltas);\n                        data_to_pyobject(py, Data::BookDeltas(deltas))\n                    })\n                    .collect::<PyResult<Vec<_>>>()\n            })\n            .map_err(|e| anyhow::anyhow!(\"Failed to convert batched deltas to Python: {e}\"));\n            Some(result)\n        }\n    }\n}\n\n#[cfg(feature = \"python\")]\n/// Streams batches of `OrderBookDeltas` Python objects from a Tardis format CSV at the given\n/// `filepath`, yielding chunks of the specified size.\n///\n/// # Errors\n///\n/// Returns an error if `chunk_size` is outside `[1, 1_000_000]`, or if the file cannot be opened,\n/// read, or parsed as CSV.\npub fn stream_batched_deltas<P: AsRef<Path>>(\n    filepath: P,\n    chunk_size: usize,\n    price_precision: Option<u8>,\n    size_precision: Option<u8>,","sourceCodeStart":565,"sourceCodeEnd":601,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/tardis/src/csv/stream.rs#L565-L601","documentation":"This Python-facing stream iterator converts each completed batch of `OrderBookDelta`s into Python objects via `data_to_pyobject` inside `Python::with_gil`. If that conversion returns a `PyResult` error (e.g. a type-conversion or interpreter failure), it is mapped into an anyhow error with this message before being yielded from `next`.","triggerScenarios":"Calling the Python `next()` on the delta stream when the Rust-to-Python conversion of a batched `OrderBookDeltas` fails — typically an error raised in Python object construction under the GIL, or a returned `PyErr` from the conversion helper.","commonSituations":"Python interpreter in a bad state (interpreter finalization during shutdown while the stream is still alive); a failing conversion for a particular instrument_id/delta payload; mixing stream objects across interpreter contexts or after `sys.exit` in scripts/notebooks.","solutions":["Ensure the stream is fully consumed (or dropped) before the Python interpreter shuts down; close iterators explicitly in `finally` blocks.","Inspect the inner `{e}` (PyErr) message for the root cause — it names the actual conversion failure.","Avoid holding stream iterators across interpreter boundaries or threads without the GIL context.","Update nautilus_trader if the failure stems from a known pyo3 conversion bug in your installed version."],"exampleFix":"// before\nfor deltas in stream:  # interpreter may finalize mid-loop\n    process(deltas)\n# after\ntry:\n    for deltas in stream:\n        process(deltas)\nfinally:\n    stream.close()  # release the Rust iterator before shutdown","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"stream = delta_stream(path, instrument_id, chunk_size)\ntry:\n    while True:\n        batch = stream.next()\n        if batch is None:\n            break\n        process(batch)\nexcept Exception as e:\n    if \"Failed to convert batched deltas to Python\" in str(e):\n        log.error(\"pyo3 conversion failed (interpreter state?): %s\", e)\n        raise\n    raise\nfinally:\n    stream.close()  # release the Rust iterator before interpreter shutdown","preventionTips":["Fully consume or explicitly close stream iterators before interpreter shutdown","Do not hold stream iterators across interpreter finalization or sys.exit paths","Keep the pyo3/nautilus_trader versions current to avoid known conversion bugs"],"tags":["rust","python","pyo3","streaming","serialization"],"backgroundTag":"json-serialization-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}