{"record":{"id":"6725bee96be10ef2","repo":"nautechsystems/nautilus_trader","slug":"cannot-convert-empty-stream-batch-to-parquet","errorCode":null,"errorMessage":"Cannot convert empty stream batch to parquet","messagePattern":"Cannot convert empty stream batch to parquet","errorType":"validation","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog.rs","lineNumber":4133,"sourceCode":"\n    fn is_record_batch_monotonic_by_ts_init(batch: &RecordBatch) -> anyhow::Result<bool> {\n        let ts_init = Self::ts_init_array(batch)?;\n        if ts_init.null_count() > 0 {\n            anyhow::bail!(\"ts_init column contains null values\");\n        }\n\n        for idx in 1..ts_init.len() {\n            if ts_init.value(idx) < ts_init.value(idx - 1) {\n                return Ok(false);\n            }\n        }\n        Ok(true)\n    }\n\n    fn ts_init_range(batch: &RecordBatch) -> anyhow::Result<(u64, u64)> {\n        let ts_init = Self::ts_init_array(batch)?;\n        if ts_init.is_empty() {\n            anyhow::bail!(\"Cannot convert empty stream batch to parquet\");\n        }\n\n        if ts_init.null_count() > 0 {\n            anyhow::bail!(\"ts_init column contains null values\");\n        }\n\n        Ok((ts_init.value(0), ts_init.value(ts_init.len() - 1)))\n    }\n\n    fn ts_init_array(batch: &RecordBatch) -> anyhow::Result<&UInt64Array> {\n        let ts_init_idx = batch\n            .schema()\n            .index_of(\"ts_init\")\n            .map_err(|_| anyhow::anyhow!(\"ts_init column not found\"))?;\n        batch\n            .column(ts_init_idx)\n            .as_any()\n            .downcast_ref::<UInt64Array>()","sourceCodeStart":4115,"sourceCodeEnd":4151,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog.rs#L4115-L4151","documentation":"`ts_init_range` computes the (min, max) ts_init interval of a batch so it can be written as a parquet file's metadata interval. An empty ts_init array means there is no data at all, and no valid interval can be derived, so the library refuses to produce a parquet file from an empty stream batch. This prevents writing empty or meaningless parquet files to the catalog.","triggerScenarios":"Calling a catalog write/consolidation path that invokes `ts_init_range` with a RecordBatch that has zero rows (e.g. a query filter matched nothing but the code still attempted a flush/write, or consolidation merged down to an empty batch).","commonSituations":"A date-range query returned no data but the caller still tried to write results to parquet; consolidation split produced an empty slice; a request filtered by a time window with no data for that instrument.","solutions":["Check the batch is non-empty (`batch.num_rows() > 0`) before invoking the write/consolidate operation.","Skip the write entirely when the query or split yielded no rows — an empty batch needs no parquet file.","Verify your query's date range and instrument identifier actually cover existing data.","If this occurs during consolidation, report it: consolidation should never produce empty batches from non-empty inputs."],"exampleFix":"// before\nlet (start, end) = ts_init_range(&batch)?;\ncatalog.write_batch(&batch)?;\n// after\nif batch.num_rows() > 0 {\n    let (start, end) = ts_init_range(&batch)?;\n    catalog.write_batch(&batch)?;\n}","handlingStrategy":"validation","validationCode":"if batch.num_rows() == 0 {\n    return Ok(()); // nothing to write — skip the parquet write entirely\n}","typeGuard":"fn is_empty_batch(batch: &arrow::record_batch::RecordBatch) -> bool {\n    batch.num_rows() == 0\n}","tryCatchPattern":"match catalog_result {\n    Err(e) if e.to_string().contains(\"empty stream batch\") => { /* skip empty write */ }\n    Err(e) => return Err(e),\n    Ok(v) => Ok(v),\n}","preventionTips":["Guard every write/consolidate call with a num_rows() > 0 check.","Filter out empty batches early when merging query results from multiple files.","Verify query date ranges actually contain data before requesting writes.","Treat empty query results as a no-op rather than a write."],"tags":["arrow","empty-data","parquet","persistence"],"backgroundTag":"empty-required-field","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}