{"record":{"id":"930cc95fb3b85e82","repo":"nautechsystems/nautilus_trader","slug":"failed-to-sort-stream-conversion-batch-e","errorCode":null,"errorMessage":"Failed to sort stream conversion batch: {e}","messagePattern":"Failed to sort stream conversion batch: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog.rs","lineNumber":4103,"sourceCode":"        }\n\n        let mut batch = concat_batches(&schema, batches.iter())\n            .map_err(|e| anyhow::anyhow!(\"Failed to concatenate stream batches: {e}\"))?;\n\n        if batch.num_rows() == 0 {\n            return Ok(None);\n        }\n\n        if !Self::is_record_batch_monotonic_by_ts_init(&batch)? {\n            let indices = sort_to_indices(\n                Self::ts_init_array(&batch)?,\n                Some(SortOptions {\n                    descending: false,\n                    nulls_first: false,\n                }),\n                None,\n            )\n            .map_err(|e| anyhow::anyhow!(\"Failed to sort stream conversion batch: {e}\"))?;\n            batch = take_record_batch(&batch, &indices)\n                .map_err(|e| anyhow::anyhow!(\"Failed to reorder stream conversion batch: {e}\"))?;\n        }\n\n        let ts_init = Self::ts_init_array(&batch)?;\n        if ts_init.null_count() > 0 {\n            anyhow::bail!(\"ts_init column contains null values\");\n        }\n\n        Ok(Some(batch))\n    }\n\n    fn is_record_batch_monotonic_by_ts_init(batch: &RecordBatch) -> anyhow::Result<bool> {\n        let ts_init = Self::ts_init_array(batch)?;\n        if ts_init.null_count() > 0 {\n            anyhow::bail!(\"ts_init column contains null values\");\n        }\n","sourceCodeStart":4085,"sourceCodeEnd":4121,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog.rs#L4085-L4121","documentation":"This error wraps a failure from Arrow's `sort_to_indices`, used when the concatenated stream batch is not monotonic by its `ts_init` column and must be sorted ascending before writing to the catalog. The sort itself almost never fails unless the `ts_init` array is in an unexpected state (e.g. contains nulls, since `nulls_first: false` sorting on nulls, or the array downcast/type is wrong) or an internal Arrow error occurs. It is surfaced with the sort operation as the named culprit.","triggerScenarios":"In `apply_stream_conversion_transforms`, after `is_record_batch_monotonic_by_ts_init` returns false (out-of-order timestamps from unordered stream messages), the code sorts `ts_init_array(&batch)`. The error appears if `ts_init_array` fails (no `ts_init` column / not UInt64 — the message then embeds that Arrow error), or the sort kernel itself errors on a null-bearing or oversized array.","commonSituations":"Stream data written with timestamps that jump backwards (clock skew, out-of-order message replay), combined with a feather file whose `ts_init` column was written with a non-UInt64 type (e.g. Int64 from an older writer) or missing entirely from a custom data type.","solutions":["Read the inner Arrow error in {e} first: if it says `ts_init column not found` or `ts_init column is not UInt64`, fix the source data schema rather than the sort.","Re-extract the stream data so messages carry valid, ordered `ts_event`/`ts_init` nanosecond timestamps.","If the data legitimately has nulls in ts_init, repair the feather file (fill ts_init from ts_event) before converting.","Convert with `use_ts_event_for_ts_init = true` if appropriate for the data type so ts_init is sourced from the always-populated ts_event column."],"exampleFix":"// before: sorting a possibly null-bearing ts_init\nlet indices = sort_to_indices(Self::ts_init_array(&batch)?, Some(SortOptions { descending: false, nulls_first: false }), None)\n    .map_err(|e| anyhow::anyhow!(\"Failed to sort stream conversion batch: {e}\"))?;\n\n// after: validate ts_init before sorting\nlet ts_init = Self::ts_init_array(&batch)?;\nif ts_init.null_count() > 0 {\n    anyhow::bail!(\"ts_init column contains null values; repair source data before sorting\");\n}\nlet indices = sort_to_indices(ts_init, Some(SortOptions { descending: false, nulls_first: false }), None)\n    .map_err(|e| anyhow::anyhow!(\"Failed to sort stream conversion batch: {e}\"))?;","handlingStrategy":"validation","validationCode":"// Rust: confirm ts_init is present, UInt64, and null-free before conversion\nfn ts_init_is_clean(batch: &RecordBatch) -> anyhow::Result<()> {\n    let idx = batch.schema().index_of(\"ts_init\")\n        .map_err(|_| anyhow::anyhow!(\"ts_init column not found\"))?;\n    let arr = batch.column(idx).as_any()\n        .downcast_ref::<UInt64Array>()\n        .ok_or_else(|| anyhow::anyhow!(\"ts_init column is not UInt64\"))?;\n    if arr.null_count() > 0 {\n        anyhow::bail!(\"ts_init column contains null values\");\n    }\n    Ok(())\n}","typeGuard":null,"tryCatchPattern":"// Rust\nmatch catalog.convert_stream_file(path) {\n    Err(e) if e.to_string().contains(\"Failed to sort stream conversion batch\") => {\n        // inspect the inner Arrow error for missing/typed ts_init\n        log::error!(\"sort failed for {path}: {e:#}\");\n    }\n    other => other?,\n}","preventionTips":["Ensure stream messages always carry monotonically sensible, populated ts_event/ts_init timestamps.","Check for clock skew / NTP issues on machines recording live data.","Pre-validate the ts_init column type (UInt64) and null count before conversion.","Sort data upstream in extraction so the conversion path never needs the sort fallback."],"tags":["arrow","rust","sorting","persistence","timestamps"],"backgroundTag":"invalid-argument-value","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}