nautechsystems/nautilus_trader · error · anyhow::Error

Failed to sort stream conversion batch: {e}

Error message

Failed to sort stream conversion batch: {e}

What it means

This error wraps a failure from Arrow's `sort_to_indices`, used when the concatenated stream batch is not monotonic by its `ts_init` column and must be sorted ascending before writing to the catalog. The sort itself almost never fails unless the `ts_init` array is in an unexpected state (e.g. contains nulls, since `nulls_first: false` sorting on nulls, or the array downcast/type is wrong) or an internal Arrow error occurs. It is surfaced with the sort operation as the named culprit.

Source

Thrown at crates/persistence/src/backend/catalog.rs:4103

        }

        let mut batch = concat_batches(&schema, batches.iter())
            .map_err(|e| anyhow::anyhow!("Failed to concatenate stream batches: {e}"))?;

        if batch.num_rows() == 0 {
            return Ok(None);
        }

        if !Self::is_record_batch_monotonic_by_ts_init(&batch)? {
            let indices = sort_to_indices(
                Self::ts_init_array(&batch)?,
                Some(SortOptions {
                    descending: false,
                    nulls_first: false,
                }),
                None,
            )
            .map_err(|e| anyhow::anyhow!("Failed to sort stream conversion batch: {e}"))?;
            batch = take_record_batch(&batch, &indices)
                .map_err(|e| anyhow::anyhow!("Failed to reorder stream conversion batch: {e}"))?;
        }

        let ts_init = Self::ts_init_array(&batch)?;
        if ts_init.null_count() > 0 {
            anyhow::bail!("ts_init column contains null values");
        }

        Ok(Some(batch))
    }

    fn is_record_batch_monotonic_by_ts_init(batch: &RecordBatch) -> anyhow::Result<bool> {
        let ts_init = Self::ts_init_array(batch)?;
        if ts_init.null_count() > 0 {
            anyhow::bail!("ts_init column contains null values");
        }

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Read the inner Arrow error in {e} first: if it says `ts_init column not found` or `ts_init column is not UInt64`, fix the source data schema rather than the sort.
  2. Re-extract the stream data so messages carry valid, ordered `ts_event`/`ts_init` nanosecond timestamps.
  3. If the data legitimately has nulls in ts_init, repair the feather file (fill ts_init from ts_event) before converting.
  4. Convert with `use_ts_event_for_ts_init = true` if appropriate for the data type so ts_init is sourced from the always-populated ts_event column.

Example fix

// before: sorting a possibly null-bearing ts_init
let indices = sort_to_indices(Self::ts_init_array(&batch)?, Some(SortOptions { descending: false, nulls_first: false }), None)
    .map_err(|e| anyhow::anyhow!("Failed to sort stream conversion batch: {e}"))?;

// after: validate ts_init before sorting
let ts_init = Self::ts_init_array(&batch)?;
if ts_init.null_count() > 0 {
    anyhow::bail!("ts_init column contains null values; repair source data before sorting");
}
let indices = sort_to_indices(ts_init, Some(SortOptions { descending: false, nulls_first: false }), None)
    .map_err(|e| anyhow::anyhow!("Failed to sort stream conversion batch: {e}"))?;
Defensive patterns

Strategy: validation

Validate before calling

// Rust: confirm ts_init is present, UInt64, and null-free before conversion
fn ts_init_is_clean(batch: &RecordBatch) -> anyhow::Result<()> {
    let idx = batch.schema().index_of("ts_init")
        .map_err(|_| anyhow::anyhow!("ts_init column not found"))?;
    let arr = batch.column(idx).as_any()
        .downcast_ref::<UInt64Array>()
        .ok_or_else(|| anyhow::anyhow!("ts_init column is not UInt64"))?;
    if arr.null_count() > 0 {
        anyhow::bail!("ts_init column contains null values");
    }
    Ok(())
}

Try / catch

// Rust
match catalog.convert_stream_file(path) {
    Err(e) if e.to_string().contains("Failed to sort stream conversion batch") => {
        // inspect the inner Arrow error for missing/typed ts_init
        log::error!("sort failed for {path}: {e:#}");
    }
    other => other?,
}

Prevention

When it happens

Trigger: In `apply_stream_conversion_transforms`, after `is_record_batch_monotonic_by_ts_init` returns false (out-of-order timestamps from unordered stream messages), the code sorts `ts_init_array(&batch)`. The error appears if `ts_init_array` fails (no `ts_init` column / not UInt64 — the message then embeds that Arrow error), or the sort kernel itself errors on a null-bearing or oversized array.

Common situations: Stream data written with timestamps that jump backwards (clock skew, out-of-order message replay), combined with a feather file whose `ts_init` column was written with a non-UInt64 type (e.g. Int64 from an older writer) or missing entirely from a custom data type.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/930cc95fb3b85e82. Report an issue: GitHub.