nautechsystems/nautilus_trader · error · anyhow::Error

Failed to reorder stream conversion batch: {e}

Error message

Failed to reorder stream conversion batch: {e}

What it means

This error wraps a failure from Arrow's `take_record_batch`, which reorders the concatenated stream batch rows according to the sort indices computed from the `ts_init` column (applied only when the batch was found non-monotonic by ts_init). `take` fails if indices are out of bounds for the batch, or the arrays in the batch are in an inconsistent/corrupt state — i.e. the sort indices no longer line up with the batch being taken.

Source

Thrown at crates/persistence/src/backend/catalog.rs:4105

        let mut batch = concat_batches(&schema, batches.iter())
            .map_err(|e| anyhow::anyhow!("Failed to concatenate stream batches: {e}"))?;

        if batch.num_rows() == 0 {
            return Ok(None);
        }

        if !Self::is_record_batch_monotonic_by_ts_init(&batch)? {
            let indices = sort_to_indices(
                Self::ts_init_array(&batch)?,
                Some(SortOptions {
                    descending: false,
                    nulls_first: false,
                }),
                None,
            )
            .map_err(|e| anyhow::anyhow!("Failed to sort stream conversion batch: {e}"))?;
            batch = take_record_batch(&batch, &indices)
                .map_err(|e| anyhow::anyhow!("Failed to reorder stream conversion batch: {e}"))?;
        }

        let ts_init = Self::ts_init_array(&batch)?;
        if ts_init.null_count() > 0 {
            anyhow::bail!("ts_init column contains null values");
        }

        Ok(Some(batch))
    }

    fn is_record_batch_monotonic_by_ts_init(batch: &RecordBatch) -> anyhow::Result<bool> {
        let ts_init = Self::ts_init_array(batch)?;
        if ts_init.null_count() > 0 {
            anyhow::bail!("ts_init column contains null values");
        }

        for idx in 1..ts_init.len() {
            if ts_init.value(idx) < ts_init.value(idx - 1) {

View on GitHub (pinned to 18893faf8b)

Solutions

1. Re-extract the offending feather file from source data; structural corruption is best fixed at the source.
2. Validate batch integrity before conversion: check all columns have equal length (`batch.num_rows() == col.len()` for each column).
3. Sort the data upstream (e.g. in the extraction script or pandas) so the batch arrives monotonic by ts_init and `take_record_batch` is never invoked.
4. Update arrow-rs to the version pinned by this nautilus release; older arrow versions had take-kernel bugs.

Example fix

// before: reorder without validating batch integrity
batch = take_record_batch(&batch, &indices)
    .map_err(|e| anyhow::anyhow!("Failed to reorder stream conversion batch: {e}"))?;

// after: verify all columns match num_rows before take
for (i, col) in batch.columns().iter().enumerate() {
    assert_eq!(col.len(), batch.num_rows(), "column {i} length mismatch");
}
batch = take_record_batch(&batch, &indices)
    .map_err(|e| anyhow::anyhow!("Failed to reorder stream conversion batch: {e}"))?;

// Rust: verify batch structural integrity (all columns equal length) before conversion
fn batch_is_consistent(batch: &RecordBatch) -> bool {
    batch.columns().iter().all(|c| c.len() == batch.num_rows())
}

// Rust
match convert(&batches) {
    Err(e) if e.to_string().contains("Failed to reorder stream conversion batch") => {
        log::error!("structural corruption in source batch: {e:#}");
        // quarantine the file and continue with the rest
    }
    other => other?,
}

When it happens

Trigger: In `apply_stream_conversion_transforms`, when the concatenated batch is non-monotonic by ts_init: `sort_to_indices` succeeds but `take_record_batch(&batch, &indices)` fails. Practically this means the record batch's arrays are inconsistent (e.g. a column length mismatch from a malformed batch, or corrupted feather data) so the gather operation cannot be performed.

Common situations: Corrupted or truncated feather stream files whose columns have inconsistent lengths; batches built by third-party tools (pandas→feather writers) that produced structurally invalid Arrow data; very rare Arrow internal errors on huge batches.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Failed to concatenate stream batches: {e}
• height must be positive, was {self.height}
• Failed to sort stream conversion batch: {e}
• ts_init column is not UInt64
• Failed to merge custom data type metadata: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/55e2631c3d438576. Report an issue: GitHub.