{"record":{"id":"97ce040bdb9ca887","repo":"nautechsystems/nautilus_trader","slug":"failed-to-create-new-batch-e-97ce04","errorCode":null,"errorMessage":"Failed to create new batch: {e}","messagePattern":"Failed to create new batch: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/custom.rs","lineNumber":258,"sourceCode":"        .first()\n        .map(arrow::array::RecordBatch::schema)\n        .ok_or_else(|| {\n            anyhow::anyhow!(\"decode_custom_batches_to_data called with empty batches\")\n        })?;\n\n    for mut batch in batches {\n        if use_ts_event_for_ts_init {\n            let column_names: Vec<String> =\n                schema.fields().iter().map(|f| f.name().clone()).collect();\n\n            if let (Some(ts_event_idx), Some(ts_init_idx)) = (\n                column_names.iter().position(|n| n == \"ts_event\"),\n                column_names.iter().position(|n| n == \"ts_init\"),\n            ) {\n                let mut new_columns = batch.columns().to_vec();\n                new_columns[ts_init_idx] = new_columns[ts_event_idx].clone();\n                batch = RecordBatch::try_new(schema.clone(), new_columns)\n                    .map_err(|e| anyhow::anyhow!(\"Failed to create new batch: {e}\"))?;\n            }\n        }\n        let metadata = batch.schema().metadata().clone();\n        let decoded = decode_batch_to_data(&metadata, batch, true)?;\n        file_data.extend(decoded);\n    }\n    Ok(file_data)\n}\n","sourceCodeStart":240,"sourceCodeEnd":267,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/custom.rs#L240-L267","documentation":"Thrown when, while substituting the ts_init column with the ts_event column (use_ts_event_for_ts_init), RecordBatch::try_new rejects the rebuilt batch. This happens if the ts_event column's Arrow type differs from the ts_init field's declared type in the schema, so the new column violates schema validation. The Arrow error is wrapped with context.","triggerScenarios":"Decoding batches with use_ts_event_for_ts_init=true where the ts_event column is not the same Arrow type as ts_init (e.g. Int64 vs UInt64, or Timestamp array).","commonSituations":"Files written by a producer that stored ts_event with a different integer type than ts_init; custom encoders with inconsistent column types; data migrated between schema versions.","solutions":["Cast the ts_event column to the ts_init type before substitution: arrow::compute::cast(&ts_event_array, ts_init_field.data_type()).","Fix the encoder so ts_event and ts_init always share the same Arrow type.","Read the wrapped {e} message to identify the exact type/length mismatch.","Only enable use_ts_event_for_ts_init for schemas known to have identical ts_event/ts_init types."],"exampleFix":"// before\nnew_columns[ts_init_idx] = new_columns[ts_event_idx].clone(); // type mismatch\n// after\nlet cast = arrow::compute::cast(&new_columns[ts_event_idx], schema.field(ts_init_idx).data_type())?;\nnew_columns[ts_init_idx] = cast;","handlingStrategy":"validation","validationCode":"anyhow::ensure!(schema.field(ts_event_idx).data_type() == schema.field(ts_init_idx).data_type(), \"ts_event/ts_init type mismatch prevents substitution\");","typeGuard":"fn ts_columns_compatible(schema: &Schema) -> bool {\n    match (schema.field_with_name(\"ts_event\"), schema.field_with_name(\"ts_init\")) {\n        (Ok(a), Ok(b)) => a.data_type() == b.data_type(),\n        _ => false,\n    }\n}","tryCatchPattern":"match decode_custom_batches_to_data(&batches, use_ts_event_for_ts_init, true) {\n    Ok(data) => use(data),\n    Err(e) if e.to_string().contains(\"Failed to create new batch\") => retry_without_ts_event_override(),\n    Err(e) => return Err(e),\n}","preventionTips":["Emit ts_event and ts_init with identical Arrow types in all encoders","Cast before column substitution when types differ","Only enable use_ts_event_for_ts_init for schemas you control","Include a decode test that exercises the ts_event override path"],"tags":["rust","arrow","persistence","type-mismatch"],"backgroundTag":"schema-validation-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}