{"record":{"id":"4f82c685ed61f69b","repo":"nautechsystems/nautilus_trader","slug":"failed-to-decode-batch-e","errorCode":null,"errorMessage":"Failed to decode batch: {e}","messagePattern":"Failed to decode batch: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog.rs","lineNumber":3891,"sourceCode":"\n                let ts_event_idx = column_names\n                    .iter()\n                    .position(|n| n == \"ts_event\")\n                    .ok_or_else(|| anyhow::anyhow!(\"ts_event column not found\"))?;\n                let ts_init_idx = column_names\n                    .iter()\n                    .position(|n| n == \"ts_init\")\n                    .ok_or_else(|| anyhow::anyhow!(\"ts_init column not found\"))?;\n\n                let mut new_columns = batch.columns().to_vec();\n                new_columns[ts_init_idx] = new_columns[ts_event_idx].clone();\n\n                batch = RecordBatch::try_new(schema.clone(), new_columns)\n                    .map_err(|e| anyhow::anyhow!(\"Failed to create new batch: {e}\"))?;\n            }\n\n            let data_vec = T::decode_data_batch(&metadata, batch)\n                .map_err(|e| anyhow::anyhow!(\"Failed to decode batch: {e}\"))?;\n\n            all_data.extend(data_vec);\n        }\n\n        Ok(to_variant::<T>(all_data))\n    }\n\n    /// Converts `RecordBatches` directly to strongly typed values.\n    fn convert_record_batches_to_typed<T>(batches: Vec<RecordBatch>) -> anyhow::Result<Vec<T>>\n    where\n        T: DecodeTypedFromRecordBatch,\n    {\n        if batches.is_empty() {\n            return Ok(Vec::new());\n        }\n\n        let mut all_data = Vec::new();\n","sourceCodeStart":3873,"sourceCodeEnd":3909,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog.rs#L3873-L3909","documentation":"Raised in `convert_record_batches_to_data_with_bar_type_conversion` when the data type's `decode_data_batch` implementation fails to turn a record batch (plus schema metadata) into domain objects. The batch reached the decoder but its schema, metadata, or contents do not match what the decoder expects for type `T`.","triggerScenarios":"Reading data with `read_run_data`/`convert_stream_to_data` where the stored batch schema or metadata (e.g. `type_name`, `bar_type`, instrument_id) does not match the decoding type `T` — wrong data_cls requested, missing metadata keys, unregistered custom type, or column type drift.","commonSituations":"Querying a data class with the wrong decoder type parameter; files written before a schema change (column renamed/retyped); bar_type metadata stored in internal format but expected external (or vice versa, when `convert_bar_type_to_external` is false); custom types not registered via `ensure_custom_data_registered::<T>()`.","solutions":["Read the wrapped message: the decoder's inner error names the exact field/metadata that failed.","Ensure the custom data type is registered before reading: call `ensure_custom_data_registered::<T>()`.","Verify the requested data_cls/decoder type `T` matches what was actually written to the files.","Re-write files produced by an older NautilusTrader version whose schema no longer matches the current decoder.","Check schema metadata (`bar_type`, `type_name`, identifier) is intact — schemaless registration paths exist to preserve it."],"exampleFix":"// before\ncatalog.ensure_custom_data_registered::<MyData>()?; // missing before read\nlet data: Vec<MyData> = read_data(...)?;\n\n// after: registration precedes decoding\ncatalog.ensure_custom_data_registered::<MyData>()?;\nlet data: Vec<MyData> = read_data(...)?;","handlingStrategy":"try-catch","validationCode":"catalog.ensure_custom_data_registered::<MyData>()?;\n// verify stored metadata keys exist before decode:\n// schema.metadata().contains_key(\"type_name\") / (\"bar_type\") as applicable","typeGuard":null,"tryCatchPattern":"match T::try_from_batches(batches) {\n    Ok(data) => data,\n    Err(e) if e.to_string().contains(\"Failed to decode batch\") => {\n        log::error!(\"schema/metadata mismatch for {}: {e}\", std::any::type_name::<T>());\n        Default::default()\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Register custom data types before both writing and reading.","Keep bar_type/instrument_id metadata intact by using schemaless registration paths.","Pin reader/writer versions per dataset; migrate files after schema changes.","Verify the generic type T matches the data_cls you requested."],"tags":["arrow","decoding","schema","persistence"],"backgroundTag":"schema-validation-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}