{"record":{"id":"cce52ddaca74ae8c","repo":"nautechsystems/nautilus_trader","slug":"record-batch-row-index-exceeds-u32","errorCode":null,"errorMessage":"record batch row index exceeds u32","messagePattern":"record batch row index exceeds u32","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/parquet.rs","lineNumber":236,"sourceCode":"    let fields: Vec<arrow_row::SortField> = schema\n        .fields()\n        .iter()\n        .map(|f| arrow_row::SortField::new(f.data_type().clone()))\n        .collect();\n\n    let converter = arrow_row::RowConverter::new(fields)?;\n    let mut seen: std::collections::HashSet<Vec<u8>> = std::collections::HashSet::new();\n    let mut result: Vec<RecordBatch> = Vec::new();\n\n    for batch in batches {\n        let rows = converter.convert_columns(batch.columns())?;\n        let mut indices: Vec<u32> = Vec::new();\n\n        for (i, row) in rows.iter().enumerate() {\n            if seen.insert(row.as_ref().to_vec()) {\n                indices.push(\n                    u32::try_from(i)\n                        .map_err(|_| anyhow::anyhow!(\"record batch row index exceeds u32\"))?,\n                );\n            }\n        }\n\n        if !indices.is_empty() {\n            let index_array = arrow::array::UInt32Array::from(indices);\n            let deduped_columns: Vec<arrow::array::ArrayRef> = batch\n                .columns()\n                .iter()\n                .map(|col| arrow::compute::take(col.as_ref(), &index_array, None))\n                .collect::<Result<_, _>>()?;\n            result.push(RecordBatch::try_new(schema.clone(), deduped_columns)?);\n        }\n    }\n\n    Ok(result)\n}\n","sourceCodeStart":218,"sourceCodeEnd":254,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/parquet.rs#L218-L254","documentation":"Thrown by deduplicate_record_batches when a retained row index cannot be represented as u32. Arrow's take/UInt32Array index arrays are limited to 2^32-1 rows, so if a record batch contains more than ~4.29 billion rows, indices no longer fit and the deduplication aborts rather than silently corrupting output.","triggerScenarios":"Deduplicating a single RecordBatch with more than u32::MAX rows — only reachable with extremely large in-memory batches assembled by combine_parquet_files_from_object_store.","commonSituations":"Combining an enormous number of parquet files into one giant batch before deduplication; memory-rich machines allowing pathological batch sizes; aggregated datasets built without row-count limits.","solutions":["Split the input into smaller batches (chunked deduplication) so each batch stays below u32::MAX rows.","Reduce the amount of data combined per call — filter by date range or partition before combining.","Upgrade to a build/version using chunked take with per-chunk offset handling if available.","Pre-deduplicate at the file level (fewer/smaller parquet files) before the combine step."],"exampleFix":"// before\nlet all = combine_parquet_files_from_object_store(store, &paths)?; // single giant batch\n// after\nfor chunk in paths.chunks(1_000) {\n    let batch = combine_parquet_files_from_object_store(store, chunk)?;\n    deduplicate_record_batches(&[batch])?;\n}","handlingStrategy":"validation","validationCode":"anyhow::ensure!(batch.num_rows() < u32::MAX as usize, \"batch too large for u32 index deduplication\");","typeGuard":"fn fits_u32(len: usize) -> bool { len < u32::MAX as usize }","tryCatchPattern":"match deduplicate_record_batches(&batches) {\n    Ok(deduped) => use(deduped),\n    Err(e) if e.to_string().contains(\"exceeds u32\") => chunked_deduplicate(&batches)?,\n    Err(e) => return Err(e),\n}","preventionTips":["Chunk combine operations so each batch stays far below u32::MAX rows","Filter by time/partition before combining parquet files","Monitor batch row counts when aggregating large catalogs","Prefer incremental deduplication over one giant batch"],"tags":["rust","arrow","persistence","limit"],"backgroundTag":"value-out-of-range","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}