nautechsystems/nautilus_trader · error · anyhow::Error

record batch row index exceeds u32

Error message

record batch row index exceeds u32

What it means

Thrown by deduplicate_record_batches when a retained row index cannot be represented as u32. Arrow's take/UInt32Array index arrays are limited to 2^32-1 rows, so if a record batch contains more than ~4.29 billion rows, indices no longer fit and the deduplication aborts rather than silently corrupting output.

Solutions

  1. Split the input into smaller batches (chunked deduplication) so each batch stays below u32::MAX rows.
  2. Reduce the amount of data combined per call — filter by date range or partition before combining.
  3. Upgrade to a build/version using chunked take with per-chunk offset handling if available.
  4. Pre-deduplicate at the file level (fewer/smaller parquet files) before the combine step.

Example fix

// before
let all = combine_parquet_files_from_object_store(store, &paths)?; // single giant batch
// after
for chunk in paths.chunks(1_000) {
    let batch = combine_parquet_files_from_object_store(store, chunk)?;
    deduplicate_record_batches(&[batch])?;
}
Defensive patterns

Strategy: validation

Validate before calling

anyhow::ensure!(batch.num_rows() < u32::MAX as usize, "batch too large for u32 index deduplication");

Type guard

fn fits_u32(len: usize) -> bool { len < u32::MAX as usize }

Try / catch

match deduplicate_record_batches(&batches) {
    Ok(deduped) => use(deduped),
    Err(e) if e.to_string().contains("exceeds u32") => chunked_deduplicate(&batches)?,
    Err(e) => return Err(e),
}

Prevention

When it happens

Trigger: Deduplicating a single RecordBatch with more than u32::MAX rows — only reachable with extremely large in-memory batches assembled by combine_parquet_files_from_object_store.

Common situations: Combining an enormous number of parquet files into one giant batch before deduplication; memory-rich machines allowing pathological batch sizes; aggregated datasets built without row-count limits.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors


AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/cce52ddaca74ae8c. Report an issue: GitHub.

Appendix: source

Thrown at crates/persistence/src/parquet.rs:236

    let fields: Vec<arrow_row::SortField> = schema
        .fields()
        .iter()
        .map(|f| arrow_row::SortField::new(f.data_type().clone()))
        .collect();

    let converter = arrow_row::RowConverter::new(fields)?;
    let mut seen: std::collections::HashSet<Vec<u8>> = std::collections::HashSet::new();
    let mut result: Vec<RecordBatch> = Vec::new();

    for batch in batches {
        let rows = converter.convert_columns(batch.columns())?;
        let mut indices: Vec<u32> = Vec::new();

        for (i, row) in rows.iter().enumerate() {
            if seen.insert(row.as_ref().to_vec()) {
                indices.push(
                    u32::try_from(i)
                        .map_err(|_| anyhow::anyhow!("record batch row index exceeds u32"))?,
                );
            }
        }

        if !indices.is_empty() {
            let index_array = arrow::array::UInt32Array::from(indices);
            let deduped_columns: Vec<arrow::array::ArrayRef> = batch
                .columns()
                .iter()
                .map(|col| arrow::compute::take(col.as_ref(), &index_array, None))
                .collect::<Result<_, _>>()?;
            result.push(RecordBatch::try_new(schema.clone(), deduped_columns)?);
        }
    }

    Ok(result)
}

View on GitHub (pinned to 18893faf8b)