{"record":{"id":"d3db03502fff28fd","repo":"nautechsystems/nautilus_trader","slug":"failed-to-call-decode-record-batch-py-e","errorCode":null,"errorMessage":"Failed to call decode_record_batch_py: {e}","messagePattern":"Failed to call decode_record_batch_py: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/model/src/python/data/mod.rs","lineNumber":478,"sourceCode":"        let pyarrow = py.import(\"pyarrow\")?;\n        let cls = pyarrow.getattr(\"RecordBatch\")?;\n        let py_batch = cls.call_method1(\n            \"_import_from_c\",\n            (\n                (&raw mut ffi_array as usize),\n                (&raw mut ffi_schema as usize),\n            ),\n        )?;\n\n        let metadata_py = pyo3::types::PyDict::new(py);\n        for (k, v) in metadata {\n            metadata_py.set_item(k, v)?;\n        }\n\n        let py_list = data_class\n            .bind(py)\n            .call_method1(\"decode_record_batch_py\", (metadata_py, py_batch))\n            .map_err(|e| anyhow::anyhow!(\"Failed to call decode_record_batch_py: {e}\"))?;\n\n        let list = py_list\n            .cast::<pyo3::types::PyList>()\n            .map_err(|_| anyhow::anyhow!(\"Expected list from decode_record_batch_py\"))?;\n\n        let mut result = Vec::new();\n        for item in list.iter() {\n            let wrapper = PythonCustomDataWrapper::new(py, &item)\n                .map_err(|e| anyhow::anyhow!(\"Failed to create wrapper: {e}\"))?;\n            result.push(crate::data::Data::Custom(\n                crate::data::CustomData::from_arc(Arc::new(wrapper)),\n            ));\n        }\n        Ok(result)\n    })\n}\n\n/// Registers a custom data **type** (class) with the catalog registry.","sourceCodeStart":460,"sourceCodeEnd":496,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/model/src/python/data/mod.rs#L460-L496","documentation":"When decoding an Arrow RecordBatch back into custom data, the decoder calls the registered Python class's `decode_record_batch_py(metadata, batch)`. This error wraps any Python exception raised inside that call; the traceback is embedded in `{e}`. The decode of the whole batch is aborted.","triggerScenarios":"Querying/decoding custom data from the catalog (via registered custom data classes) when `decode_record_batch_py` raises — e.g. the batch schema does not match what the class expects, columns are missing or renamed, or the metadata dict lacks expected keys (common after Parquet/DataFusion drops schema metadata).","commonSituations":"Parquet files written by an older schema version read with a newer class definition; DataFusion round-trips that lose Arrow schema metadata so the decoder can't recover type information; hand-written decoders that assume non-nullable columns receiving nulls.","solutions":["Read the embedded Python traceback and fix the exception inside `decode_record_batch_py`.","Verify the stored Parquet schema matches the currently registered class schema (column names, types, nullability).","Re-register the type with `ensure_custom_data_registered::<T>()` so schema metadata is present, and rewrite old data if the schema changed.","Handle nulls/missing columns defensively in the decoder implementation."],"exampleFix":"# before\ndef decode_record_batch_py(cls, metadata, batch):\n    return [cls(x[\"price\"], x[\"qty\"]) for x in batch]\n\n# after\ndef decode_record_batch_py(cls, metadata, batch):\n    return [cls(x.get(\"price\"), x.get(\"qty\")) for x in batch]  # tolerate missing/null columns","handlingStrategy":"try-catch","validationCode":"required = {f.name for f in registered_schema}\nstored_names = set(batch.schema.names)\nassert required <= stored_names, f\"missing columns: {required - stored_names}\"","typeGuard":null,"tryCatchPattern":"try:\n    items = decode_record_batch(batch, MyData)\nexcept Exception as e:\n    logger.error(f\"decode_record_batch_py raised: {e}\")\n    raise","preventionTips":["Call ensure_custom_data_registered::<T>() before reading data","Handle nulls and absent columns in decode_record_batch_py","Version your schema and migrate stored Parquet when fields change"],"tags":["python","arrow","custom-data","decoding","parquet"],"backgroundTag":"schema-validation-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}