{"record":{"id":"ac888950392e838e","repo":"nautechsystems/nautilus_trader","slug":"intervals-are-not-disjoint-after-consolidating-a-d","errorCode":null,"errorMessage":"Intervals are not disjoint after consolidating a directory","messagePattern":"Intervals are not disjoint after consolidating a directory","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog_operations.rs","lineNumber":341,"sourceCode":"                .iter()\n                .map(|path| ObjectPath::from(path.as_str()))\n                .collect();\n\n            self.execute_async(async {\n                combine_parquet_files_from_object_store(\n                    self.object_store.clone(),\n                    object_paths,\n                    &ObjectPath::from(path),\n                    Some(self.compression),\n                    Some(self.max_row_group_size),\n                    deduplicate,\n                )\n                .await\n            })?;\n        }\n\n        if ensure_contiguous_files.unwrap_or(true) && !are_intervals_disjoint(&intervals) {\n            anyhow::bail!(\"Intervals are not disjoint after consolidating a directory\");\n        }\n\n        Ok(())\n    }\n\n    /// Consolidates all data files in the catalog by splitting them into fixed time periods.\n    ///\n    /// This method identifies all leaf directories in the catalog that contain parquet files\n    /// and consolidates them by period. A leaf directory is one that contains files but no subdirectories.\n    /// This is a convenience method that effectively calls `consolidate_data_by_period` for all data types\n    /// and instrument IDs in the catalog.\n    ///\n    /// # Parameters\n    ///\n    /// - `period_nanos`: The period duration for consolidation in nanoseconds. Default is 1 day (86400000000000).\n    ///   Examples: 3600000000000 (1 hour), 604800000000000 (7 days), 1800000000000 (30 minutes)\n    /// - `start`: Optional start timestamp for the consolidation range. Only files with timestamps\n    ///   greater than or equal to this value will be consolidated. If None, all files","sourceCodeStart":323,"sourceCodeEnd":359,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog_operations.rs#L323-L359","documentation":"After `consolidate_directory` merges/splits the parquet files in a directory, it recomputes every file's [start, end] ts_init interval and, when `ensure_contiguous_files` is enabled (the default), asserts the intervals are mutually disjoint. If overlapping intervals remain, the consolidation did not produce a clean, contiguous layout and the operation aborts to prevent ambiguous time-range lookups. This is an internal post-condition check on the catalog's file layout invariant.","triggerScenarios":"Running `consolidate_catalog` or `consolidate_data` (which call `consolidate_directory`) where, after the consolidation pass, two or more files in the directory still have overlapping ts_init ranges — e.g. data written concurrently during consolidation, files outside the consolidated period that overlap, or duplicate/overlapping writes to the same instrument directory.","commonSituations":"A live strategy kept writing to the catalog while consolidation ran in another process/thread; pre-existing overlapping files from earlier crashes or manual copies; running two consolidation jobs at once; files whose periods overlap because an earlier consolidation used a different period setting.","solutions":["Stop all writers and re-run consolidation so no concurrent writes create overlapping files.","Set `ensure_contiguous_files: Some(false)` if overlapping files are acceptable for your workload, or pre-split consolidation into non-overlapping windows.","Inspect the instrument directory for duplicate/overlapping parquet files (same instrument, overlapping ts_init ranges) and delete or re-write them.","Run consolidation single-threaded / with an exclusive lock on the catalog directory."],"exampleFix":"// before: consolidating while live writers are active\ncatalog.consolidate_catalog(None).await?;\n// after: opt out of the contiguity requirement or ensure exclusive access\ncatalog.consolidate_catalog_url(None, Some(false)).await?; // ensure_contiguous_files = false","handlingStrategy":"try-catch","validationCode":"// Before consolidating, ensure no other process/thread writes to the catalog dir\n// and that existing files are disjoint:\n// fn files_disjoint(intervals: &[(u64, u64)]) -> bool {\n//     let mut s = intervals.clone(); s.sort();\n//     s.windows(2).all(|w| w[0].1 < w[1].0)\n// }","typeGuard":null,"tryCatchPattern":"match catalog.consolidate_catalog_url(None, None).await {\n    Err(e) if e.to_string().contains(\"Intervals are not disjoint\") => {\n        // stop writers, inspect overlapping files, retry consolidation exclusively\n    }\n    Err(e) => return Err(e),\n    Ok(()) => Ok(()),\n}","preventionTips":["Pause all live writers or snapshot the catalog before running consolidation.","Run only one consolidation job at a time; use a lock or single scheduler.","Periodically verify file intervals are disjoint with a small audit script.","Pass ensure_contiguous_files=false if your workload tolerates overlapping files."],"tags":["consolidation","parquet","concurrency","data-integrity"],"backgroundTag":"invalid-state-transition","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}