nautechsystems/nautilus_trader · error · anyhow::Error

ts_init column not found

Error message

ts_init column not found

What it means

Raised in `convert_record_batches_to_data_with_bar_type_conversion` when `use_ts_event_for_ts_init` is true but the record-batch schema has no `ts_init` column, so the destination column of the `ts_event → ts_init` copy cannot be found. Decoding expects `ts_init` to be present, so the transform cannot proceed.

Source

Thrown at crates/persistence/src/backend/catalog.rs:3881

        if convert_bar_type_to_external {
            convert_bar_type_metadata_to_external(&mut metadata);
        }

        let mut all_data = Vec::new();

        for mut batch in batches {
            if use_ts_event_for_ts_init {
                let column_names: Vec<String> =
                    schema.fields().iter().map(|f| f.name().clone()).collect();

                let ts_event_idx = column_names
                    .iter()
                    .position(|n| n == "ts_event")
                    .ok_or_else(|| anyhow::anyhow!("ts_event column not found"))?;
                let ts_init_idx = column_names
                    .iter()
                    .position(|n| n == "ts_init")
                    .ok_or_else(|| anyhow::anyhow!("ts_init column not found"))?;

                let mut new_columns = batch.columns().to_vec();
                new_columns[ts_init_idx] = new_columns[ts_event_idx].clone();

                batch = RecordBatch::try_new(schema.clone(), new_columns)
                    .map_err(|e| anyhow::anyhow!("Failed to create new batch: {e}"))?;
            }

            let data_vec = T::decode_data_batch(&metadata, batch)
                .map_err(|e| anyhow::anyhow!("Failed to decode batch: {e}"))?;

            all_data.extend(data_vec);
        }

        Ok(to_variant::<T>(all_data))
    }

    /// Converts `RecordBatches` directly to strongly typed values.

View on GitHub (pinned to 18893faf8b)

Solutions

1. Inspect the file's schema field names and confirm `ts_init` is present; re-write the file with the current writer if not.
2. Disable the transform by calling the reader with `use_ts_event_for_ts_init=false` when the data type does not need the ts_event→ts_init copy.
3. Correct the data_cls or identifier filter so only files with the expected schema are converted.
4. If the files are custom data, re-register the type with a schema containing `ts_init` and rewrite the data.

fn has_required_ts_columns(schema: &Schema) -> bool {
    schema.field_with_name("ts_event").is_ok() && schema.field_with_name("ts_init").is_ok()
}

When it happens

Trigger: Same conversion paths as the `ts_event` variant: enabling `use_ts_event_for_ts_init` on batches whose schema lacks `ts_init` — legacy files, custom data without timestamp columns, or a mismatched data_cls selecting the wrong files.

Common situations: Reading old stream files written before `ts_init` was standardized; hand-crafted or externally produced Arrow files missing the `ts_init` field; requesting the conversion for a data type whose decoder schema omits `ts_init`.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• ts_event column not found
• height must be positive, was {self.height}
• Failed to decode batch: {e}
• Failed to concatenate stream batches: {e}
• Failed to merge custom data type metadata: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/4ea7f314adca2e45. Report an issue: GitHub.