nautechsystems/nautilus_trader · error · anyhow::Error

ts_event column not found

Error message

ts_event column not found

What it means

Raised in `convert_record_batches_to_data_with_bar_type_conversion` when `use_ts_event_for_ts_init` is true but the record-batch schema has no `ts_event` column. The transform needs to copy `ts_event` into `ts_init` before decoding, so it aborts when the required source column is absent.

Source

Thrown at crates/persistence/src/backend/catalog.rs:3877

        let schema = batches[0].schema();
        let mut metadata = schema.metadata().clone();

        if convert_bar_type_to_external {
            convert_bar_type_metadata_to_external(&mut metadata);
        }

        let mut all_data = Vec::new();

        for mut batch in batches {
            if use_ts_event_for_ts_init {
                let column_names: Vec<String> =
                    schema.fields().iter().map(|f| f.name().clone()).collect();

                let ts_event_idx = column_names
                    .iter()
                    .position(|n| n == "ts_event")
                    .ok_or_else(|| anyhow::anyhow!("ts_event column not found"))?;
                let ts_init_idx = column_names
                    .iter()
                    .position(|n| n == "ts_init")
                    .ok_or_else(|| anyhow::anyhow!("ts_init column not found"))?;

                let mut new_columns = batch.columns().to_vec();
                new_columns[ts_init_idx] = new_columns[ts_event_idx].clone();

                batch = RecordBatch::try_new(schema.clone(), new_columns)
                    .map_err(|e| anyhow::anyhow!("Failed to create new batch: {e}"))?;
            }

            let data_vec = T::decode_data_batch(&metadata, batch)
                .map_err(|e| anyhow::anyhow!("Failed to decode batch: {e}"))?;

            all_data.extend(data_vec);
        }

View on GitHub (pinned to 18893faf8b)

Solutions

1. Call the conversion with `use_ts_event_for_ts_init=false` for data types whose schema has no `ts_event` column.
2. Inspect the batch schema (print the field names) and confirm which timestamp columns the file actually contains.
3. Verify the correct data_cls/identifier is requested so the matching files with `ts_event` are selected.
4. Re-write legacy files with the current writer so the schema includes `ts_event` if the flag is required.
5. If it is custom data, ensure the registered Arrow schema matches what was written (see `ensure_custom_data_registered::<T>()`).

Example fix

// before
catalog.convert_stream_to_data(instance_id, "my_custom_data", None, None, true)?;

// after: this type has no ts_event column
catalog.convert_stream_to_data(instance_id, "my_custom_data", None, None, false)?;

fn has_ts_event(schema: &Schema) -> bool { schema.field_with_name("ts_event").is_ok() }
// only pass use_ts_event_for_ts_init=true when this holds for the file's schema

When it happens

Trigger: Calling data-reading paths (e.g. `convert_stream_to_data` / `read_run_data` variants) with `use_ts_event_for_ts_init=true` against batches whose schema lacks `ts_event` — typically non-market data types, custom data, or files written by a schema variant without timestamp columns.

Common situations: Enabling the ts_event-for-ts_init flag for a data class that never stored `ts_event`; reading legacy feather/parquet files with an older column set; passing the wrong data_cls so an unexpected file set is read; custom data types that only carry `ts_init`.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• ts_init column not found
• height must be positive, was {self.height}
• Failed to decode batch: {e}
• Failed to concatenate stream batches: {e}
• Failed to merge custom data type metadata: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/364821327c88bf70. Report an issue: GitHub.