nautechsystems/nautilus_trader · error · anyhow::Error

Failed to call decode_record_batch_py: {e}

Error message

Failed to call decode_record_batch_py: {e}

What it means

When decoding an Arrow RecordBatch back into custom data, the decoder calls the registered Python class's `decode_record_batch_py(metadata, batch)`. This error wraps any Python exception raised inside that call; the traceback is embedded in `{e}`. The decode of the whole batch is aborted.

Source

Thrown at crates/model/src/python/data/mod.rs:478

        let pyarrow = py.import("pyarrow")?;
        let cls = pyarrow.getattr("RecordBatch")?;
        let py_batch = cls.call_method1(
            "_import_from_c",
            (
                (&raw mut ffi_array as usize),
                (&raw mut ffi_schema as usize),
            ),
        )?;

        let metadata_py = pyo3::types::PyDict::new(py);
        for (k, v) in metadata {
            metadata_py.set_item(k, v)?;
        }

        let py_list = data_class
            .bind(py)
            .call_method1("decode_record_batch_py", (metadata_py, py_batch))
            .map_err(|e| anyhow::anyhow!("Failed to call decode_record_batch_py: {e}"))?;

        let list = py_list
            .cast::<pyo3::types::PyList>()
            .map_err(|_| anyhow::anyhow!("Expected list from decode_record_batch_py"))?;

        let mut result = Vec::new();
        for item in list.iter() {
            let wrapper = PythonCustomDataWrapper::new(py, &item)
                .map_err(|e| anyhow::anyhow!("Failed to create wrapper: {e}"))?;
            result.push(crate::data::Data::Custom(
                crate::data::CustomData::from_arc(Arc::new(wrapper)),
            ));
        }
        Ok(result)
    })
}

/// Registers a custom data **type** (class) with the catalog registry.

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Read the embedded Python traceback and fix the exception inside `decode_record_batch_py`.
  2. Verify the stored Parquet schema matches the currently registered class schema (column names, types, nullability).
  3. Re-register the type with `ensure_custom_data_registered::<T>()` so schema metadata is present, and rewrite old data if the schema changed.
  4. Handle nulls/missing columns defensively in the decoder implementation.

Example fix

# before
def decode_record_batch_py(cls, metadata, batch):
    return [cls(x["price"], x["qty"]) for x in batch]

# after
def decode_record_batch_py(cls, metadata, batch):
    return [cls(x.get("price"), x.get("qty")) for x in batch]  # tolerate missing/null columns
Defensive patterns

Strategy: try-catch

Validate before calling

required = {f.name for f in registered_schema}
stored_names = set(batch.schema.names)
assert required <= stored_names, f"missing columns: {required - stored_names}"

Try / catch

try:
    items = decode_record_batch(batch, MyData)
except Exception as e:
    logger.error(f"decode_record_batch_py raised: {e}")
    raise

Prevention

When it happens

Trigger: Querying/decoding custom data from the catalog (via registered custom data classes) when `decode_record_batch_py` raises — e.g. the batch schema does not match what the class expects, columns are missing or renamed, or the metadata dict lacks expected keys (common after Parquet/DataFusion drops schema metadata).

Common situations: Parquet files written by an older schema version read with a newer class definition; DataFusion round-trips that lose Arrow schema metadata so the decoder can't recover type information; hand-written decoders that assume non-nullable columns receiving nulls.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/d3db03502fff28fd. Report an issue: GitHub.