nautechsystems/nautilus_trader · error · anyhow::Error

Failed to call encode_record_batch_py: {e}

Error message

Failed to call encode_record_batch_py: {e}

What it means

When the first custom data item exposes the Python method `encode_record_batch_py`, the encoder calls it with the list of Python items to produce a RecordBatch-like object. This error wraps any Python exception raised during that call, including errors raised inside the user class's `encode_record_batch_py` implementation. The Python traceback text is embedded in `{e}`.

Source

Thrown at crates/model/src/python/data/mod.rs:399

        let py_items: Result<Vec<_>, _> = items.iter().map(|item| item.to_pyobject(py)).collect();
        let py_items = py_items.map_err(|e| anyhow::anyhow!("Failed to convert to Python: {e}"))?;
        let py_list = pyo3::types::PyList::new(py, &py_items)
            .map_err(|e| anyhow::anyhow!("Failed to create list: {e}"))?;

        let first = items
            .first()
            .ok_or_else(|| anyhow::anyhow!("No items to encode"))?;
        let first_py = first.to_pyobject(py)?;

        if first_py
            .bind(py)
            .hasattr("encode_record_batch_py")
            .unwrap_or(false)
        {
            let py_batch = first_py
                .bind(py)
                .call_method1("encode_record_batch_py", (py_list,))
                .map_err(|e| anyhow::anyhow!("Failed to call encode_record_batch_py: {e}"))?;

            let mut ffi_array = arrow::ffi::FFI_ArrowArray::empty();
            let mut ffi_schema = arrow::ffi::FFI_ArrowSchema::empty();

            py_batch.call_method1(
                "_export_to_c",
                (
                    (&raw mut ffi_array as usize),
                    (&raw mut ffi_schema as usize),
                ),
            )?;

            let schema = std::sync::Arc::new(arrow::datatypes::Schema::try_from(&ffi_schema)?);
            let struct_array_data = unsafe {
                arrow::ffi::from_ffi_and_data_type(
                    ffi_array,
                    arrow::datatypes::DataType::Struct(schema.fields().clone()),
                )?

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Read the embedded Python traceback and fix the exception inside `encode_record_batch_py`.
  2. Ensure every item in the list is an instance of the same class with the same schema/fields.
  3. Test `MyClass.encode_record_batch_py([instance])` directly in Python with a representative item.
  4. If relying on generated encoders, regenerate them after any change to the data class fields.
Defensive patterns

Strategy: try-catch

Validate before calling

assert len({type(x) for x in items}) == 1, "mixed item classes in batch"
batch_out = MyClass.encode_record_batch_py(items)  # smoke-test in Python first

Try / catch

try:
    batch = encode_to_record_batch(items)
except Exception as e:
    logger.error(f"encode_record_batch_py raised: {e}")
    raise

Prevention

When it happens

Trigger: Encoding registered custom data to Arrow when the Python class's `encode_record_batch_py(items)` raises — e.g. schema mismatch between items, missing columns, wrong item types in the list, or a bug in the user's encode implementation.

Common situations: User-defined data class registered with a hand-written `encode_record_batch_py` that assumes all items share identical fields; mixed item versions after a schema change so old and new objects are encoded together; typo in column names vs the declared Arrow schema.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/7812950304c7c874. Report an issue: GitHub.