{"record":{"id":"364821327c88bf70","repo":"nautechsystems/nautilus_trader","slug":"ts-event-column-not-found","errorCode":null,"errorMessage":"ts_event column not found","messagePattern":"ts_event column not found","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog.rs","lineNumber":3877,"sourceCode":"\n        let schema = batches[0].schema();\n        let mut metadata = schema.metadata().clone();\n\n        if convert_bar_type_to_external {\n            convert_bar_type_metadata_to_external(&mut metadata);\n        }\n\n        let mut all_data = Vec::new();\n\n        for mut batch in batches {\n            if use_ts_event_for_ts_init {\n                let column_names: Vec<String> =\n                    schema.fields().iter().map(|f| f.name().clone()).collect();\n\n                let ts_event_idx = column_names\n                    .iter()\n                    .position(|n| n == \"ts_event\")\n                    .ok_or_else(|| anyhow::anyhow!(\"ts_event column not found\"))?;\n                let ts_init_idx = column_names\n                    .iter()\n                    .position(|n| n == \"ts_init\")\n                    .ok_or_else(|| anyhow::anyhow!(\"ts_init column not found\"))?;\n\n                let mut new_columns = batch.columns().to_vec();\n                new_columns[ts_init_idx] = new_columns[ts_event_idx].clone();\n\n                batch = RecordBatch::try_new(schema.clone(), new_columns)\n                    .map_err(|e| anyhow::anyhow!(\"Failed to create new batch: {e}\"))?;\n            }\n\n            let data_vec = T::decode_data_batch(&metadata, batch)\n                .map_err(|e| anyhow::anyhow!(\"Failed to decode batch: {e}\"))?;\n\n            all_data.extend(data_vec);\n        }\n","sourceCodeStart":3859,"sourceCodeEnd":3895,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog.rs#L3859-L3895","documentation":"Raised in `convert_record_batches_to_data_with_bar_type_conversion` when `use_ts_event_for_ts_init` is true but the record-batch schema has no `ts_event` column. The transform needs to copy `ts_event` into `ts_init` before decoding, so it aborts when the required source column is absent.","triggerScenarios":"Calling data-reading paths (e.g. `convert_stream_to_data` / `read_run_data` variants) with `use_ts_event_for_ts_init=true` against batches whose schema lacks `ts_event` — typically non-market data types, custom data, or files written by a schema variant without timestamp columns.","commonSituations":"Enabling the ts_event-for-ts_init flag for a data class that never stored `ts_event`; reading legacy feather/parquet files with an older column set; passing the wrong data_cls so an unexpected file set is read; custom data types that only carry `ts_init`.","solutions":["Call the conversion with `use_ts_event_for_ts_init=false` for data types whose schema has no `ts_event` column.","Inspect the batch schema (print the field names) and confirm which timestamp columns the file actually contains.","Verify the correct data_cls/identifier is requested so the matching files with `ts_event` are selected.","Re-write legacy files with the current writer so the schema includes `ts_event` if the flag is required.","If it is custom data, ensure the registered Arrow schema matches what was written (see `ensure_custom_data_registered::<T>()`)."],"exampleFix":"// before\ncatalog.convert_stream_to_data(instance_id, \"my_custom_data\", None, None, true)?;\n\n// after: this type has no ts_event column\ncatalog.convert_stream_to_data(instance_id, \"my_custom_data\", None, None, false)?;","handlingStrategy":"validation","validationCode":"fn has_ts_event(schema: &Schema) -> bool { schema.field_with_name(\"ts_event\").is_ok() }\n// only pass use_ts_event_for_ts_init=true when this holds for the file's schema","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Only enable use_ts_event_for_ts_init for market data types whose schema includes ts_event.","Inspect file schemas (field names) before bulk conversion.","Keep custom data schemas registered with the standard ts_event/ts_init columns.","After a library upgrade, spot-check a sample file's schema before converting folders."],"tags":["arrow","schema","timestamp","persistence"],"backgroundTag":"schema-validation-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}