nautechsystems/nautilus_trader · error

Failed to convert to Python: {e}

Error message

Failed to convert to Python: {e}

What it means

When encoding custom data items to an Arrow RecordBatch, each Rust-side item (`Arc<dyn CustomDataTrait>`) is first converted to its Python representation via `to_pyobject`. This error is raised if any item fails that Rust→Python conversion; the underlying pyo3/Python error is embedded in `{e}`. It aborts the whole batch encode — no partial output is produced.

Source

Thrown at crates/model/src/python/data/mod.rs:382

            .call_method1("from_json", (py_dict,))
            .map_err(|e| anyhow::anyhow!("Failed to call from_json: {e}"))?;

        let wrapper = PythonCustomDataWrapper::new(py, &instance)
            .map_err(|e| anyhow::anyhow!("Failed to create wrapper: {e}"))?;

        Ok(Arc::new(wrapper) as Arc<dyn crate::data::CustomDataTrait>)
    })
}

/// Encodes `CustomData` items to `RecordBatch` via Python `encode_record_batch_py`.
#[allow(unsafe_code)]
#[cfg(all(feature = "python", feature = "arrow"))]
fn py_encode_custom_data_to_record_batch(
    items: &[std::sync::Arc<dyn crate::data::CustomDataTrait>],
) -> Result<arrow::record_batch::RecordBatch, anyhow::Error> {
    pyo3::Python::attach(|py| {
        let py_items: Result<Vec<_>, _> = items.iter().map(|item| item.to_pyobject(py)).collect();
        let py_items = py_items.map_err(|e| anyhow::anyhow!("Failed to convert to Python: {e}"))?;
        let py_list = pyo3::types::PyList::new(py, &py_items)
            .map_err(|e| anyhow::anyhow!("Failed to create list: {e}"))?;

        let first = items
            .first()
            .ok_or_else(|| anyhow::anyhow!("No items to encode"))?;
        let first_py = first.to_pyobject(py)?;

        if first_py
            .bind(py)
            .hasattr("encode_record_batch_py")
            .unwrap_or(false)
        {
            let py_batch = first_py
                .bind(py)
                .call_method1("encode_record_batch_py", (py_list,))
                .map_err(|e| anyhow::anyhow!("Failed to call encode_record_batch_py: {e}"))?;

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Read the embedded `{e}` message to identify which item and why the conversion failed.
  2. Ensure all items in the batch are live, valid instances of the registered custom data class before encoding.
  3. Re-create the items (decode again from source) rather than reusing wrappers across interpreter lifecycles.
  4. Encode items in smaller batches to isolate the failing item.
Defensive patterns

Strategy: validation

Validate before calling

assert all(isinstance(x, MyClass) for x in items), "all items must be live registered-class instances"

Try / catch

try:
    batch = encode_to_record_batch(items)
except Exception as e:
    logger.error(f"item -> Python conversion failed: {e}")
    raise

Prevention

When it happens

Trigger: Calling the Arrow encode path for registered custom data (via `register_custom_data_class`-registered types) when one or more items in the slice fail `to_pyobject(py)` — typically because the wrapper holds a dead/invalid Python reference or the item's conversion method raises.

Common situations: Items were decoded from a stream whose Python objects were garbage-collected or the interpreter state changed; a custom wrapper's `to_pyobject` implementation raises; mixing items whose underlying Python class no longer matches the registered class after hot-reload.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/7814200d41d572d5. Report an issue: GitHub.