nautechsystems/nautilus_trader · error · anyhow::Error

ts_init column contains null values

Error message

ts_init column contains null values

What it means

After sorting/reordering a stream-conversion record batch by ts_init, `apply_stream_conversion_transforms` asserts that the ts_init column has no nulls, since ts_init is the primary ordering/partitioning key for parquet writes. Null ts_init values would break monotonic ordering and interval partitioning, so the transform bails.

Source

Thrown at crates/persistence/src/backend/catalog.rs:4110

        }

        if !Self::is_record_batch_monotonic_by_ts_init(&batch)? {
            let indices = sort_to_indices(
                Self::ts_init_array(&batch)?,
                Some(SortOptions {
                    descending: false,
                    nulls_first: false,
                }),
                None,
            )
            .map_err(|e| anyhow::anyhow!("Failed to sort stream conversion batch: {e}"))?;
            batch = take_record_batch(&batch, &indices)
                .map_err(|e| anyhow::anyhow!("Failed to reorder stream conversion batch: {e}"))?;
        }

        let ts_init = Self::ts_init_array(&batch)?;
        if ts_init.null_count() > 0 {
            anyhow::bail!("ts_init column contains null values");
        }

        Ok(Some(batch))
    }

    fn is_record_batch_monotonic_by_ts_init(batch: &RecordBatch) -> anyhow::Result<bool> {
        let ts_init = Self::ts_init_array(batch)?;
        if ts_init.null_count() > 0 {
            anyhow::bail!("ts_init column contains null values");
        }

        for idx in 1..ts_init.len() {
            if ts_init.value(idx) < ts_init.value(idx - 1) {
                return Ok(false);
            }
        }
        Ok(true)
    }

View on GitHub (pinned to 18893faf8b)

Solutions

1. Clean the source feather data: drop or repair rows with null ts_init before conversion.
2. Backfill ts_init with sensible values (e.g. derived from the record's timestamp) in a preprocessing step.
3. Exclude the offending feather files from conversion and re-capture or regenerate them from a trustworthy source.

// Before conversion: verify ts_init is non-null in the feather table
let ts_init_col = table.column_by_name("ts_init").expect("ts_init column");
assert_eq!(ts_init_col.null_count(), 0, "feather data has null ts_init; clean it first");

match result {
    Err(e) if e.to_string().contains("ts_init column contains null") => {
        log::warn!("stream has null ts_init rows; drop/repair these rows before converting");
    }
    other => other?,
}

When it happens

Trigger: Running feather→parquet conversion on a stream whose feather records contain missing/null ts_init values — e.g. corrupted or truncated feather rows, records written by older writers without ts_init, or schema drift leaving the ts_init column unset for some rows.

Common situations: Legacy feather files produced before ts_init was mandatory; partially written/corrupted capture sessions; manually edited feather data where rows were appended without timestamps.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• Replacement hash {transaction_hash} conflicts with another i
• Included wrap transaction {tx_hash} has invalid block number
• WETH balance overflow for included transaction {tx_hash} at
• conflicting RTDS TWAP observation topic={} symbol={} timesta
• Duplicate fill event for position {position_id}: {}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/15483a57adf155a0. Report an issue: GitHub.