nautechsystems/nautilus_trader · error · anyhow::Error

Failed to create new batch: {e}

Error message

Failed to create new batch: {e}

What it means

Thrown when, while substituting the ts_init column with the ts_event column (use_ts_event_for_ts_init), RecordBatch::try_new rejects the rebuilt batch. This happens if the ts_event column's Arrow type differs from the ts_init field's declared type in the schema, so the new column violates schema validation. The Arrow error is wrapped with context.

Source

Thrown at crates/persistence/src/backend/custom.rs:258

        .first()
        .map(arrow::array::RecordBatch::schema)
        .ok_or_else(|| {
            anyhow::anyhow!("decode_custom_batches_to_data called with empty batches")
        })?;

    for mut batch in batches {
        if use_ts_event_for_ts_init {
            let column_names: Vec<String> =
                schema.fields().iter().map(|f| f.name().clone()).collect();

            if let (Some(ts_event_idx), Some(ts_init_idx)) = (
                column_names.iter().position(|n| n == "ts_event"),
                column_names.iter().position(|n| n == "ts_init"),
            ) {
                let mut new_columns = batch.columns().to_vec();
                new_columns[ts_init_idx] = new_columns[ts_event_idx].clone();
                batch = RecordBatch::try_new(schema.clone(), new_columns)
                    .map_err(|e| anyhow::anyhow!("Failed to create new batch: {e}"))?;
            }
        }
        let metadata = batch.schema().metadata().clone();
        let decoded = decode_batch_to_data(&metadata, batch, true)?;
        file_data.extend(decoded);
    }
    Ok(file_data)
}

View on GitHub (pinned to 18893faf8b)

Solutions

1. Cast the ts_event column to the ts_init type before substitution: arrow::compute::cast(&ts_event_array, ts_init_field.data_type()).
2. Fix the encoder so ts_event and ts_init always share the same Arrow type.
3. Read the wrapped {e} message to identify the exact type/length mismatch.
4. Only enable use_ts_event_for_ts_init for schemas known to have identical ts_event/ts_init types.

Example fix

// before
new_columns[ts_init_idx] = new_columns[ts_event_idx].clone(); // type mismatch
// after
let cast = arrow::compute::cast(&new_columns[ts_event_idx], schema.field(ts_init_idx).data_type())?;
new_columns[ts_init_idx] = cast;

anyhow::ensure!(schema.field(ts_event_idx).data_type() == schema.field(ts_init_idx).data_type(), "ts_event/ts_init type mismatch prevents substitution");

fn ts_columns_compatible(schema: &Schema) -> bool {
    match (schema.field_with_name("ts_event"), schema.field_with_name("ts_init")) {
        (Ok(a), Ok(b)) => a.data_type() == b.data_type(),
        _ => false,
    }
}

match decode_custom_batches_to_data(&batches, use_ts_event_for_ts_init, true) {
    Ok(data) => use(data),
    Err(e) if e.to_string().contains("Failed to create new batch") => retry_without_ts_event_override(),
    Err(e) => return Err(e),
}

When it happens

Trigger: Decoding batches with use_ts_event_for_ts_init=true where the ts_event column is not the same Arrow type as ts_init (e.g. Int64 vs UInt64, or Timestamp array).

Common situations: Files written by a producer that stored ts_event with a different integer type than ts_init; custom encoders with inconsistent column types; data migrated between schema versions.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• ts_init column is not UInt64
• Expected {}, was different type
• Expected {}
• Expected Custom data variant
• Failed to concatenate stream batches: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/97ce040bdb9ca887. Report an issue: GitHub.