nautechsystems/nautilus_trader · error · anyhow::Error

Failed to apply stream conversion transforms for {feather_pa

Error message

Failed to apply stream conversion transforms for {feather_path}: {e}

What it means

Raised in `convert_feather_batches_to_parquet` when `apply_stream_conversion_transforms` fails while converting a feather stream file's batches before writing them to the catalog as parquet. The original transform error (e.g. the ts_event/ts_init column or batch-rebuild errors above) is wrapped together with the source feather path for context.

Source

Thrown at crates/persistence/src/backend/catalog.rs:3985

            )?;
        }

        Ok(())
    }

    fn convert_feather_batches_to_parquet(
        &self,
        data_name: &str,
        feather_path: &str,
        batches: Vec<RecordBatch>,
        use_ts_event_for_ts_init: bool,
    ) -> anyhow::Result<()> {
        let Some(batch) = Self::apply_stream_conversion_transforms(
            batches,
            use_ts_event_for_ts_init,
        )
        .map_err(|e| {
            anyhow::anyhow!("Failed to apply stream conversion transforms for {feather_path}: {e}")
        })?
        else {
            return Ok(());
        };

        let (start_ts, end_ts) = Self::ts_init_range(&batch).map_err(|e| {
            anyhow::anyhow!("Failed to determine ts_init range for {feather_path}: {e}")
        })?;
        let identifier = Self::identifier_from_batch_or_path(&batch, data_name, feather_path);
        let directory = if let Some(type_name) = data_name.strip_prefix("custom/") {
            self.make_path_custom_data(type_name, identifier.as_deref())?
        } else {
            self.make_path(data_name, identifier.as_deref())?
        };
        let filename = timestamps_to_filename(UnixNanos::from(start_ts), UnixNanos::from(end_ts));
        let path = PathBuf::from(format!("{directory}/{filename}"));
        let object_path = self.to_object_path(&path.to_string_lossy())?;

View on GitHub (pinned to 18893faf8b)

Solutions

1. Identify the failing file from the `{feather_path}` in the message and inspect its schema field names/types.
2. Fix the root cause reported by the inner transform error (missing ts_event/ts_init column or type mismatch), typically by re-writing that file with the current writer.
3. Convert with `use_ts_event_for_ts_init=false` if the data type does not require the ts_init replacement.
4. Exclude or relocate files whose schema is incompatible, then re-run the conversion for the rest of the folder.
5. Regenerate the whole stream folder from the original run data if many files predate a schema change.

Example fix

// before
catalog.convert_stream_to_data(instance_id, "quotes", Some("backtest"), None, true)?;

// after: quotes file lacks ts_event; skip the ts_init substitution
catalog.convert_stream_to_data(instance_id, "quotes", Some("backtest"), None, false)?;

// pre-flight per file: schema has ts_event and ts_init with identical Arrow types
fn convertible(schema: &Schema) -> bool {
    schema.field_with_name("ts_event").is_ok()
        && schema.field_with_name("ts_init").is_ok()
        && schema.field_with_name("ts_event").unwrap().data_type()
            == schema.field_with_name("ts_init").unwrap().data_type()
}

match convert_stream_to_data_checked(instance_id, data_cls, subdirectory, None, use_ts_event_for_ts_init) {
    Ok(()) => (),
    Err(e) if e.to_string().contains("stream conversion transforms") => {
        // path is embedded in the message; quarantine that file and continue
        let path = extract_feather_path(&e.to_string());
        quarantine(&path);
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling `convert_stream_to_data` with `use_ts_event_for_ts_init=true` (or a batch set needing bar-type conversion) where the per-file batch transform fails: missing `ts_event`/`ts_init` columns, Arrow type mismatch when copying columns, or metadata conversion failing for that specific file.

Common situations: Migrating legacy backtest stream folders to the parquet catalog where some files have old schemas; mixed-schema files within one data class directory so one file fails while others convert; custom or externally written feather files lacking the standard timestamp columns.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• height must be positive, was {self.height}
• Cannot convert empty stream batch to parquet
• Failed to call decode_record_batch_py: {e}
• e.to_string()
• Failed to create StreamReader: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/753e39bd2046c40c. Report an issue: GitHub.