nautechsystems/nautilus_trader · error · anyhow::Error

Failed to decode batch: {e}

Error message

Failed to decode batch: {e}

What it means

Raised in `convert_record_batches_to_data_with_bar_type_conversion` when the data type's `decode_data_batch` implementation fails to turn a record batch (plus schema metadata) into domain objects. The batch reached the decoder but its schema, metadata, or contents do not match what the decoder expects for type `T`.

Source

Thrown at crates/persistence/src/backend/catalog.rs:3891

                let ts_event_idx = column_names
                    .iter()
                    .position(|n| n == "ts_event")
                    .ok_or_else(|| anyhow::anyhow!("ts_event column not found"))?;
                let ts_init_idx = column_names
                    .iter()
                    .position(|n| n == "ts_init")
                    .ok_or_else(|| anyhow::anyhow!("ts_init column not found"))?;

                let mut new_columns = batch.columns().to_vec();
                new_columns[ts_init_idx] = new_columns[ts_event_idx].clone();

                batch = RecordBatch::try_new(schema.clone(), new_columns)
                    .map_err(|e| anyhow::anyhow!("Failed to create new batch: {e}"))?;
            }

            let data_vec = T::decode_data_batch(&metadata, batch)
                .map_err(|e| anyhow::anyhow!("Failed to decode batch: {e}"))?;

            all_data.extend(data_vec);
        }

        Ok(to_variant::<T>(all_data))
    }

    /// Converts `RecordBatches` directly to strongly typed values.
    fn convert_record_batches_to_typed<T>(batches: Vec<RecordBatch>) -> anyhow::Result<Vec<T>>
    where
        T: DecodeTypedFromRecordBatch,
    {
        if batches.is_empty() {
            return Ok(Vec::new());
        }

        let mut all_data = Vec::new();

View on GitHub (pinned to 18893faf8b)

Solutions

1. Read the wrapped message: the decoder's inner error names the exact field/metadata that failed.
2. Ensure the custom data type is registered before reading: call `ensure_custom_data_registered::<T>()`.
3. Verify the requested data_cls/decoder type `T` matches what was actually written to the files.
4. Re-write files produced by an older NautilusTrader version whose schema no longer matches the current decoder.
5. Check schema metadata (`bar_type`, `type_name`, identifier) is intact — schemaless registration paths exist to preserve it.

Example fix

// before
catalog.ensure_custom_data_registered::<MyData>()?; // missing before read
let data: Vec<MyData> = read_data(...)?;

// after: registration precedes decoding
catalog.ensure_custom_data_registered::<MyData>()?;
let data: Vec<MyData> = read_data(...)?;

catalog.ensure_custom_data_registered::<MyData>()?;
// verify stored metadata keys exist before decode:
// schema.metadata().contains_key("type_name") / ("bar_type") as applicable

match T::try_from_batches(batches) {
    Ok(data) => data,
    Err(e) if e.to_string().contains("Failed to decode batch") => {
        log::error!("schema/metadata mismatch for {}: {e}", std::any::type_name::<T>());
        Default::default()
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Reading data with `read_run_data`/`convert_stream_to_data` where the stored batch schema or metadata (e.g. `type_name`, `bar_type`, instrument_id) does not match the decoding type `T` — wrong data_cls requested, missing metadata keys, unregistered custom type, or column type drift.

Common situations: Querying a data class with the wrong decoder type parameter; files written before a schema change (column renamed/retyped); bar_type metadata stored in internal format but expected external (or vice versa, when `convert_bar_type_to_external` is false); custom types not registered via `ensure_custom_data_registered::<T>()`.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• height must be positive, was {self.height}
• ts_event column not found
• ts_init column not found
• Failed to concatenate stream batches: {e}
• Failed to merge custom data type metadata: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/4f82c685ed61f69b. Report an issue: GitHub.