{"record":{"id":"aad0f088636bb2b4","repo":"nautechsystems/nautilus_trader","slug":"failed-to-concatenate-stream-batches-e","errorCode":null,"errorMessage":"Failed to concatenate stream batches: {e}","messagePattern":"Failed to concatenate stream batches: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog.rs","lineNumber":4088,"sourceCode":"\n            for batch in &mut batches {\n                let mut columns = batch.columns().to_vec();\n                columns[ts_init_idx] = columns[ts_event_idx].clone();\n\n                *batch = RecordBatch::try_new(schema.clone(), columns).map_err(|e| {\n                    anyhow::anyhow!(\"Failed to create stream conversion batch: {e}\")\n                })?;\n            }\n        } else if metadata_changed {\n            for batch in &mut batches {\n                *batch = RecordBatch::try_new(schema.clone(), batch.columns().to_vec()).map_err(\n                    |e| anyhow::anyhow!(\"Failed to create stream conversion batch: {e}\"),\n                )?;\n            }\n        }\n\n        let mut batch = concat_batches(&schema, batches.iter())\n            .map_err(|e| anyhow::anyhow!(\"Failed to concatenate stream batches: {e}\"))?;\n\n        if batch.num_rows() == 0 {\n            return Ok(None);\n        }\n\n        if !Self::is_record_batch_monotonic_by_ts_init(&batch)? {\n            let indices = sort_to_indices(\n                Self::ts_init_array(&batch)?,\n                Some(SortOptions {\n                    descending: false,\n                    nulls_first: false,\n                }),\n                None,\n            )\n            .map_err(|e| anyhow::anyhow!(\"Failed to sort stream conversion batch: {e}\"))?;\n            batch = take_record_batch(&batch, &indices)\n                .map_err(|e| anyhow::anyhow!(\"Failed to reorder stream conversion batch: {e}\"))?;\n        }","sourceCodeStart":4070,"sourceCodeEnd":4106,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog.rs#L4070-L4106","documentation":"This error wraps an Arrow `concat_batches` failure while `apply_stream_conversion_transforms` merges all per-message RecordBatches from a feather/nautilus stream file into one batch before writing it to the catalog as parquet. `concat_batches` fails when the input batches cannot be combined under a single schema — typically because a batch's schema differs from `batches[0].schema()` (column order, field types, or metadata-driven field changes). It is an upstream data-shape problem, not an I/O failure.","triggerScenarios":"Called from stream-to-catalog conversion (e.g. `convert_catalog` / feather import path) when the Vec<RecordBatch> read from a `.feather` stream file contains batches with heterogeneous schemas — e.g. mixed arrow versions wrote the file, or the ts_event→ts_init substitution (`use_ts_event_for_ts_init`) rebuilt batches against a schema that other batches no longer match, or the bar_type metadata rewrite produced a schema inconsistent with later batches.","commonSituations":"Converting stream/feather data written by an older nautilus version into a newer catalog where the Arrow schema for a data type changed; hand-edited or partially-written feather files; a stream directory mixing data files of different schema versions; corrupted feather files read as batches with mismatched fields.","solutions":["Re-extract or re-generate the source feather/stream files so all batches share one schema (re-run the stream session or re-export with the current nautilus version).","Check the Arrow error text in {e}: it names the exact schema mismatch (field order/type); if it is a column-order difference, normalize batch schemas before conversion with `RecordBatch::try_new(schema, ...)` per batch.","Convert one data-type directory at a time to isolate which file contains the mismatched batch, then exclude/fix that file.","Upgrade both the writer and reader sides of the data to matching nautilus/arrow versions so serialization formats agree."],"exampleFix":"// before: concatenating raw heterogeneous batches\nlet batch = concat_batches(&schema, batches.iter())\n    .map_err(|e| anyhow::anyhow!(\"Failed to concatenate stream batches: {e}\"))?;\n\n// after: normalize each batch to the first batch's schema first\nlet schema = batches[0].schema();\nlet batches: Vec<RecordBatch> = batches\n    .into_iter()\n    .map(|b| {\n        if b.schema() != schema {\n            RecordBatch::try_new(schema.clone(), b.columns().to_vec())\n                .expect(\"batch normalized to common schema\")\n        } else {\n            b\n        }\n    })\n    .collect();\nlet batch = concat_batches(&schema, batches.iter())\n    .map_err(|e| anyhow::anyhow!(\"Failed to concatenate stream batches: {e}\"))?;","handlingStrategy":"validation","validationCode":"// Rust: verify all batches share an identical schema before conversion\nfn schemas_match(batches: &[RecordBatch]) -> bool {\n    if batches.is_empty() { return false; }\n    let first = batches[0].schema();\n    batches.iter().all(|b| b.schema() == first)\n}\nif !schemas_match(&batches) {\n    return Err(anyhow::anyhow!(\"source batches have heterogeneous schemas; re-extract data\"));\n}","typeGuard":null,"tryCatchPattern":"// Rust\nmatch convert_stream_to_catalog(&feather_path) {\n    Ok(()) => info!(\"converted {feather_path}\"),\n    Err(e) if e.to_string().contains(\"Failed to concatenate stream batches\") => {\n        warn!(\"skipping {feather_path}: schema mismatch ({e})\");\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Write all stream/feather data with one consistent nautilusTrader version per dataset directory.","Never mix feather files from different writer versions in one conversion run.","Validate batch schemas right after reading feather files, before any transform.","Keep a checksum or schema fingerprint of datasets to detect mixed-version data early."],"tags":["arrow","rust","data-conversion","schema","persistence"],"backgroundTag":"schema-validation-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}