{"record":{"id":"15483a57adf155a0","repo":"nautechsystems/nautilus_trader","slug":"ts-init-column-contains-null-values","errorCode":null,"errorMessage":"ts_init column contains null values","messagePattern":"ts_init column contains null values","errorType":"validation","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog.rs","lineNumber":4110,"sourceCode":"        }\n\n        if !Self::is_record_batch_monotonic_by_ts_init(&batch)? {\n            let indices = sort_to_indices(\n                Self::ts_init_array(&batch)?,\n                Some(SortOptions {\n                    descending: false,\n                    nulls_first: false,\n                }),\n                None,\n            )\n            .map_err(|e| anyhow::anyhow!(\"Failed to sort stream conversion batch: {e}\"))?;\n            batch = take_record_batch(&batch, &indices)\n                .map_err(|e| anyhow::anyhow!(\"Failed to reorder stream conversion batch: {e}\"))?;\n        }\n\n        let ts_init = Self::ts_init_array(&batch)?;\n        if ts_init.null_count() > 0 {\n            anyhow::bail!(\"ts_init column contains null values\");\n        }\n\n        Ok(Some(batch))\n    }\n\n    fn is_record_batch_monotonic_by_ts_init(batch: &RecordBatch) -> anyhow::Result<bool> {\n        let ts_init = Self::ts_init_array(batch)?;\n        if ts_init.null_count() > 0 {\n            anyhow::bail!(\"ts_init column contains null values\");\n        }\n\n        for idx in 1..ts_init.len() {\n            if ts_init.value(idx) < ts_init.value(idx - 1) {\n                return Ok(false);\n            }\n        }\n        Ok(true)\n    }","sourceCodeStart":4092,"sourceCodeEnd":4128,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog.rs#L4092-L4128","documentation":"After sorting/reordering a stream-conversion record batch by ts_init, `apply_stream_conversion_transforms` asserts that the ts_init column has no nulls, since ts_init is the primary ordering/partitioning key for parquet writes. Null ts_init values would break monotonic ordering and interval partitioning, so the transform bails.","triggerScenarios":"Running feather→parquet conversion on a stream whose feather records contain missing/null ts_init values — e.g. corrupted or truncated feather rows, records written by older writers without ts_init, or schema drift leaving the ts_init column unset for some rows.","commonSituations":"Legacy feather files produced before ts_init was mandatory; partially written/corrupted capture sessions; manually edited feather data where rows were appended without timestamps.","solutions":["Clean the source feather data: drop or repair rows with null ts_init before conversion.","Backfill ts_init with sensible values (e.g. derived from the record's timestamp) in a preprocessing step.","Exclude the offending feather files from conversion and re-capture or regenerate them from a trustworthy source."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before conversion: verify ts_init is non-null in the feather table\nlet ts_init_col = table.column_by_name(\"ts_init\").expect(\"ts_init column\");\nassert_eq!(ts_init_col.null_count(), 0, \"feather data has null ts_init; clean it first\");","typeGuard":null,"tryCatchPattern":"match result {\n    Err(e) if e.to_string().contains(\"ts_init column contains null\") => {\n        log::warn!(\"stream has null ts_init rows; drop/repair these rows before converting\");\n    }\n    other => other?,\n}","preventionTips":["Sanity-check each feather file for nulls in required columns (ts_init, ts_event) as part of a pre-migration audit.","Never hand-edit feather files; regenerate them from source data instead.","Detect and quarantine corrupted/truncated capture files early in the migration pipeline."],"tags":["parquet","data-integrity","null-values","rust"],"backgroundTag":"empty-required-field","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}