{"record":{"id":"9d9392345f1b64f2","repo":"nautechsystems/nautilus_trader","slug":"intervals-are-not-contiguous-when-ensure-contiguo","errorCode":null,"errorMessage":"Intervals are not contiguous. When ensure_contiguous_files=true, all files in the consolidation range must have contiguous timestamps.","messagePattern":"Intervals are not contiguous\\. When ensure_contiguous_files=true, all files in the consolidation range must have contiguous timestamps\\.","errorType":"validation","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog_operations.rs","lineNumber":1282,"sourceCode":"\n        let mut filtered_intervals = Vec::new();\n\n        for &(interval_start, interval_end) in intervals {\n            // Check if interval overlaps with the specified range\n            if used_start.is_none_or(|used_start| used_start <= interval_end)\n                && used_end.is_none_or(|used_end| interval_start <= used_end)\n            {\n                filtered_intervals.push((interval_start, interval_end));\n            }\n        }\n\n        if filtered_intervals.is_empty() {\n            return Ok(Vec::new()); // No intervals in the specified range\n        }\n\n        // Check contiguity of filtered intervals if required\n        if ensure_contiguous_files && !are_intervals_contiguous(&filtered_intervals) {\n            anyhow::bail!(\n                \"Intervals are not contiguous. When ensure_contiguous_files=true, \\\n                 all files in the consolidation range must have contiguous timestamps.\"\n            );\n        }\n\n        // Group intervals by the target period: split only when the gap between files\n        // exceeds one period, since sub-period gaps land in the same consolidated file.\n        let contiguous_groups = self.group_contiguous_intervals(&filtered_intervals, period_nanos);\n\n        let mut queries_to_execute = Vec::new();\n\n        // Handle interval splitting by creating split operations for data preservation\n        if !filtered_intervals.is_empty() {\n            if let Some(start_ts) = used_start {\n                let first_interval = filtered_intervals[0];\n                if first_interval.0 < start_ts && start_ts <= first_interval.1 {\n                    // Split before start: preserve data from interval_start to start-1\n                    queries_to_execute.push(ConsolidationQuery {","sourceCodeStart":1264,"sourceCodeEnd":1300,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog_operations.rs#L1264-L1300","documentation":"Raised by prepare_consolidation_queries when consolidation is requested with ensure_contiguous_files=true but the Parquet files (intervals) in the target range have gaps between their timestamps. The library requires that consolidation only merge files whose intervals form a contiguous sequence, otherwise the merged file would misrepresent data availability. This is a data-integrity guard for the catalog's file layout.","triggerScenarios":"Calling consolidate_data_by_period_generic or consolidate_custom_data_by_period with ensure_contiguous_files=true while the range contains files with non-contiguous start/end timestamps (e.g. missing sessions, deleted files in the middle of the range).","commonSituations":"Ranges spanning holidays/weekends where no data was recorded, files deleted individually for backfill or storage reclamation, ingestion that skipped periods due to feed outages, or manually copied catalog directories with partial data.","solutions":["Set ensure_contiguous_files=false if gaps are acceptable and consolidation need not preserve contiguity.","Narrow the consolidation range so it only includes files with contiguous timestamps.","Backfill or regenerate the missing data files so intervals become contiguous, then re-run consolidation.","Inspect the catalog intervals (e.g. via get_directory_intervals) to identify which files have gaps before retrying."],"exampleFix":"// before\nlet queries = catalog.prepare_consolidation_queries(\n    &identifier, data_type, start, end, /* ensure_contiguous_files */ true,\n)?;\n// after — allow gaps in the range\nlet queries = catalog.prepare_consolidation_queries(\n    &identifier, data_type, start, end, /* ensure_contiguous_files */ false,\n)?;","handlingStrategy":"validation","validationCode":"// check contiguity before requesting consolidation\nlet intervals = catalog.get_directory_intervals(&directory)?;\nif ensure_contiguous && !are_intervals_contiguous(&intervals) {\n    // narrow the range or backfill gaps first\n    return Err(\"range has gaps; consolidation would produce non-contiguous file\".into());\n}","typeGuard":null,"tryCatchPattern":"match catalog.consolidate_data_by_period_generic(...) {\n    Err(e) if e.to_string().contains(\"Intervals are not contiguous\") => {\n        // retry with ensure_contiguous_files=false or backfill missing files\n    }\n    other => other?,\n}","preventionTips":["Never delete individual files from the middle of a contiguous catalog range without re-checking contiguity.","Backfill gaps immediately after feed outages before consolidating.","Keep ensure_contiguous_files=true only for ranges you know are fully ingested."],"tags":["rust","persistence","data-integrity","consolidation"],"backgroundTag":"invalid-state-transition","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}