nautechsystems/nautilus_trader · error · anyhow::Error

Cannot convert empty stream batch to parquet

Error message

Cannot convert empty stream batch to parquet

What it means

`ts_init_range` computes the (min, max) ts_init interval of a batch so it can be written as a parquet file's metadata interval. An empty ts_init array means there is no data at all, and no valid interval can be derived, so the library refuses to produce a parquet file from an empty stream batch. This prevents writing empty or meaningless parquet files to the catalog.

Source

Thrown at crates/persistence/src/backend/catalog.rs:4133

    fn is_record_batch_monotonic_by_ts_init(batch: &RecordBatch) -> anyhow::Result<bool> {
        let ts_init = Self::ts_init_array(batch)?;
        if ts_init.null_count() > 0 {
            anyhow::bail!("ts_init column contains null values");
        }

        for idx in 1..ts_init.len() {
            if ts_init.value(idx) < ts_init.value(idx - 1) {
                return Ok(false);
            }
        }
        Ok(true)
    }

    fn ts_init_range(batch: &RecordBatch) -> anyhow::Result<(u64, u64)> {
        let ts_init = Self::ts_init_array(batch)?;
        if ts_init.is_empty() {
            anyhow::bail!("Cannot convert empty stream batch to parquet");
        }

        if ts_init.null_count() > 0 {
            anyhow::bail!("ts_init column contains null values");
        }

        Ok((ts_init.value(0), ts_init.value(ts_init.len() - 1)))
    }

    fn ts_init_array(batch: &RecordBatch) -> anyhow::Result<&UInt64Array> {
        let ts_init_idx = batch
            .schema()
            .index_of("ts_init")
            .map_err(|_| anyhow::anyhow!("ts_init column not found"))?;
        batch
            .column(ts_init_idx)
            .as_any()
            .downcast_ref::<UInt64Array>()

View on GitHub (pinned to 18893faf8b)

Solutions

1. Check the batch is non-empty (`batch.num_rows() > 0`) before invoking the write/consolidate operation.
2. Skip the write entirely when the query or split yielded no rows — an empty batch needs no parquet file.
3. Verify your query's date range and instrument identifier actually cover existing data.
4. If this occurs during consolidation, report it: consolidation should never produce empty batches from non-empty inputs.

Example fix

// before
let (start, end) = ts_init_range(&batch)?;
catalog.write_batch(&batch)?;
// after
if batch.num_rows() > 0 {
    let (start, end) = ts_init_range(&batch)?;
    catalog.write_batch(&batch)?;
}

if batch.num_rows() == 0 {
    return Ok(()); // nothing to write — skip the parquet write entirely
}

fn is_empty_batch(batch: &arrow::record_batch::RecordBatch) -> bool {
    batch.num_rows() == 0
}

match catalog_result {
    Err(e) if e.to_string().contains("empty stream batch") => { /* skip empty write */ }
    Err(e) => return Err(e),
    Ok(v) => Ok(v),
}

When it happens

Trigger: Calling a catalog write/consolidation path that invokes `ts_init_range` with a RecordBatch that has zero rows (e.g. a query filter matched nothing but the code still attempted a flush/write, or consolidation merged down to an empty batch).

Common situations: A date-range query returned no data but the caller still tried to write results to parquet; consolidation split produced an empty slice; a request filtered by a time window with no data for that instrument.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• height must be positive, was {self.height}
• e.to_string()
• Unsupported Data::Defi variant for catalog writes
• Unsupported Data variant for catalog writes
• Cannot write {type_name} data with mixed identities: element

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/6725bee96be10ef2. Report an issue: GitHub.