{"record":{"id":"753e39bd2046c40c","repo":"nautechsystems/nautilus_trader","slug":"failed-to-apply-stream-conversion-transforms-for","errorCode":null,"errorMessage":"Failed to apply stream conversion transforms for {feather_path}: {e}","messagePattern":"Failed to apply stream conversion transforms for (.+?): (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog.rs","lineNumber":3985,"sourceCode":"            )?;\n        }\n\n        Ok(())\n    }\n\n    fn convert_feather_batches_to_parquet(\n        &self,\n        data_name: &str,\n        feather_path: &str,\n        batches: Vec<RecordBatch>,\n        use_ts_event_for_ts_init: bool,\n    ) -> anyhow::Result<()> {\n        let Some(batch) = Self::apply_stream_conversion_transforms(\n            batches,\n            use_ts_event_for_ts_init,\n        )\n        .map_err(|e| {\n            anyhow::anyhow!(\"Failed to apply stream conversion transforms for {feather_path}: {e}\")\n        })?\n        else {\n            return Ok(());\n        };\n\n        let (start_ts, end_ts) = Self::ts_init_range(&batch).map_err(|e| {\n            anyhow::anyhow!(\"Failed to determine ts_init range for {feather_path}: {e}\")\n        })?;\n        let identifier = Self::identifier_from_batch_or_path(&batch, data_name, feather_path);\n        let directory = if let Some(type_name) = data_name.strip_prefix(\"custom/\") {\n            self.make_path_custom_data(type_name, identifier.as_deref())?\n        } else {\n            self.make_path(data_name, identifier.as_deref())?\n        };\n        let filename = timestamps_to_filename(UnixNanos::from(start_ts), UnixNanos::from(end_ts));\n        let path = PathBuf::from(format!(\"{directory}/{filename}\"));\n        let object_path = self.to_object_path(&path.to_string_lossy())?;\n","sourceCodeStart":3967,"sourceCodeEnd":4003,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog.rs#L3967-L4003","documentation":"Raised in `convert_feather_batches_to_parquet` when `apply_stream_conversion_transforms` fails while converting a feather stream file's batches before writing them to the catalog as parquet. The original transform error (e.g. the ts_event/ts_init column or batch-rebuild errors above) is wrapped together with the source feather path for context.","triggerScenarios":"Calling `convert_stream_to_data` with `use_ts_event_for_ts_init=true` (or a batch set needing bar-type conversion) where the per-file batch transform fails: missing `ts_event`/`ts_init` columns, Arrow type mismatch when copying columns, or metadata conversion failing for that specific file.","commonSituations":"Migrating legacy backtest stream folders to the parquet catalog where some files have old schemas; mixed-schema files within one data class directory so one file fails while others convert; custom or externally written feather files lacking the standard timestamp columns.","solutions":["Identify the failing file from the `{feather_path}` in the message and inspect its schema field names/types.","Fix the root cause reported by the inner transform error (missing ts_event/ts_init column or type mismatch), typically by re-writing that file with the current writer.","Convert with `use_ts_event_for_ts_init=false` if the data type does not require the ts_init replacement.","Exclude or relocate files whose schema is incompatible, then re-run the conversion for the rest of the folder.","Regenerate the whole stream folder from the original run data if many files predate a schema change."],"exampleFix":"// before\ncatalog.convert_stream_to_data(instance_id, \"quotes\", Some(\"backtest\"), None, true)?;\n\n// after: quotes file lacks ts_event; skip the ts_init substitution\ncatalog.convert_stream_to_data(instance_id, \"quotes\", Some(\"backtest\"), None, false)?;","handlingStrategy":"try-catch","validationCode":"// pre-flight per file: schema has ts_event and ts_init with identical Arrow types\nfn convertible(schema: &Schema) -> bool {\n    schema.field_with_name(\"ts_event\").is_ok()\n        && schema.field_with_name(\"ts_init\").is_ok()\n        && schema.field_with_name(\"ts_event\").unwrap().data_type()\n            == schema.field_with_name(\"ts_init\").unwrap().data_type()\n}","typeGuard":null,"tryCatchPattern":"match convert_stream_to_data_checked(instance_id, data_cls, subdirectory, None, use_ts_event_for_ts_init) {\n    Ok(()) => (),\n    Err(e) if e.to_string().contains(\"stream conversion transforms\") => {\n        // path is embedded in the message; quarantine that file and continue\n        let path = extract_feather_path(&e.to_string());\n        quarantine(&path);\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Run a schema pre-check on every feather file before folder-wide conversion.","Convert one file first as a canary, then batch the remainder.","Handle per-file failures by quarantining and continuing rather than aborting the migration.","Regenerate legacy stream folders from raw data after major version upgrades."],"tags":["arrow","feather","parquet","migration"],"backgroundTag":"schema-validation-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}