{"record":{"id":"55e2631c3d438576","repo":"nautechsystems/nautilus_trader","slug":"failed-to-reorder-stream-conversion-batch-e","errorCode":null,"errorMessage":"Failed to reorder stream conversion batch: {e}","messagePattern":"Failed to reorder stream conversion batch: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog.rs","lineNumber":4105,"sourceCode":"        let mut batch = concat_batches(&schema, batches.iter())\n            .map_err(|e| anyhow::anyhow!(\"Failed to concatenate stream batches: {e}\"))?;\n\n        if batch.num_rows() == 0 {\n            return Ok(None);\n        }\n\n        if !Self::is_record_batch_monotonic_by_ts_init(&batch)? {\n            let indices = sort_to_indices(\n                Self::ts_init_array(&batch)?,\n                Some(SortOptions {\n                    descending: false,\n                    nulls_first: false,\n                }),\n                None,\n            )\n            .map_err(|e| anyhow::anyhow!(\"Failed to sort stream conversion batch: {e}\"))?;\n            batch = take_record_batch(&batch, &indices)\n                .map_err(|e| anyhow::anyhow!(\"Failed to reorder stream conversion batch: {e}\"))?;\n        }\n\n        let ts_init = Self::ts_init_array(&batch)?;\n        if ts_init.null_count() > 0 {\n            anyhow::bail!(\"ts_init column contains null values\");\n        }\n\n        Ok(Some(batch))\n    }\n\n    fn is_record_batch_monotonic_by_ts_init(batch: &RecordBatch) -> anyhow::Result<bool> {\n        let ts_init = Self::ts_init_array(batch)?;\n        if ts_init.null_count() > 0 {\n            anyhow::bail!(\"ts_init column contains null values\");\n        }\n\n        for idx in 1..ts_init.len() {\n            if ts_init.value(idx) < ts_init.value(idx - 1) {","sourceCodeStart":4087,"sourceCodeEnd":4123,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog.rs#L4087-L4123","documentation":"This error wraps a failure from Arrow's `take_record_batch`, which reorders the concatenated stream batch rows according to the sort indices computed from the `ts_init` column (applied only when the batch was found non-monotonic by ts_init). `take` fails if indices are out of bounds for the batch, or the arrays in the batch are in an inconsistent/corrupt state — i.e. the sort indices no longer line up with the batch being taken.","triggerScenarios":"In `apply_stream_conversion_transforms`, when the concatenated batch is non-monotonic by ts_init: `sort_to_indices` succeeds but `take_record_batch(&batch, &indices)` fails. Practically this means the record batch's arrays are inconsistent (e.g. a column length mismatch from a malformed batch, or corrupted feather data) so the gather operation cannot be performed.","commonSituations":"Corrupted or truncated feather stream files whose columns have inconsistent lengths; batches built by third-party tools (pandas→feather writers) that produced structurally invalid Arrow data; very rare Arrow internal errors on huge batches.","solutions":["Re-extract the offending feather file from source data; structural corruption is best fixed at the source.","Validate batch integrity before conversion: check all columns have equal length (`batch.num_rows() == col.len()` for each column).","Sort the data upstream (e.g. in the extraction script or pandas) so the batch arrives monotonic by ts_init and `take_record_batch` is never invoked.","Update arrow-rs to the version pinned by this nautilus release; older arrow versions had take-kernel bugs."],"exampleFix":"// before: reorder without validating batch integrity\nbatch = take_record_batch(&batch, &indices)\n    .map_err(|e| anyhow::anyhow!(\"Failed to reorder stream conversion batch: {e}\"))?;\n\n// after: verify all columns match num_rows before take\nfor (i, col) in batch.columns().iter().enumerate() {\n    assert_eq!(col.len(), batch.num_rows(), \"column {i} length mismatch\");\n}\nbatch = take_record_batch(&batch, &indices)\n    .map_err(|e| anyhow::anyhow!(\"Failed to reorder stream conversion batch: {e}\"))?;","handlingStrategy":"validation","validationCode":"// Rust: verify batch structural integrity (all columns equal length) before conversion\nfn batch_is_consistent(batch: &RecordBatch) -> bool {\n    batch.columns().iter().all(|c| c.len() == batch.num_rows())\n}","typeGuard":null,"tryCatchPattern":"// Rust\nmatch convert(&batches) {\n    Err(e) if e.to_string().contains(\"Failed to reorder stream conversion batch\") => {\n        log::error!(\"structural corruption in source batch: {e:#}\");\n        // quarantine the file and continue with the rest\n    }\n    other => other?,\n}","preventionTips":["Validate feather file integrity (equal-length columns) before conversion.","Avoid third-party feather writers that may produce structurally invalid Arrow data.","Pre-sort data by ts_init in the extraction step so the take/reorder path is skipped.","Keep arrow-rs and nautilusTrader versions aligned; update both together."],"tags":["arrow","rust","data-conversion","persistence","record-batch"],"backgroundTag":"internal-invariant-violation","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}