{"record":{"id":"8fc37f811d50611a","repo":"nautechsystems/nautilus_trader","slug":"writing-file-filename-with-interval-start-ts","errorCode":null,"errorMessage":"Writing file {filename} with interval ({start_ts}, {end_ts}) would create non-disjoint intervals. Existing intervals: {current_intervals:?}","messagePattern":"Writing file (.+?) with interval \\((.+?), (.+?)\\) would create non-disjoint intervals\\. Existing intervals: (.+?)","errorType":"validation","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/catalog.rs","lineNumber":646,"sourceCode":"\n        let file_exists = self.execute_async(async {\n            let exists: bool = self.object_store.head(&object_path).await.is_ok();\n            Ok(exists)\n        })?;\n\n        if file_exists {\n            log::info!(\"File {} already exists, skipping write\", path.display());\n            return Ok(path);\n        }\n\n        if !skip_disjoint_check.unwrap_or(false) {\n            let current_intervals = self.get_directory_intervals(&directory)?;\n            let new_interval = (start_ts.as_u64(), end_ts.as_u64());\n            let mut new_intervals = current_intervals.clone();\n            new_intervals.push(new_interval);\n\n            if !are_intervals_disjoint(&new_intervals) {\n                anyhow::bail!(\n                    \"Writing file {filename} with interval ({start_ts}, {end_ts}) would create \\\n                    non-disjoint intervals. Existing intervals: {current_intervals:?}\"\n                );\n            }\n        }\n\n        log::info!(\n            \"Writing {} batches of {type_name} data to {}\",\n            batches.len(),\n            path.display(),\n        );\n\n        self.execute_async(async {\n            write_batches_to_object_store(\n                &batches,\n                self.object_store.clone(),\n                &object_path,\n                Some(self.compression),","sourceCodeStart":628,"sourceCodeEnd":664,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/catalog.rs#L628-L664","documentation":"`write_to_parquet` maintains non-overlapping (disjoint) time intervals per data directory. Before writing, it collects existing file intervals in the target directory and rejects the write if the new file's `(ts_start, ts_end)` would overlap any of them, because overlapping files make reads ambiguous.","triggerScenarios":"Writing the same data type into the same catalog directory twice with overlapping time ranges — e.g. re-running an ingestion job, re-recording a session into the same catalog, or appending data whose timestamps fall inside an already-written file's range.","commonSituations":"Re-running a recording script without clearing the catalog; ingesting overlapping historical chunks; a live recorder restarted while old files remain in the directory.","solutions":["Delete or move the existing overlapping files in the catalog directory before rewriting.","Use a fresh catalog directory or session name for each recording run.","Write only the time range not already covered, or use `extend_file_name` to append to the existing file instead.","Check existing intervals first via the catalog's directory interval listing and trim the new batch accordingly."],"exampleFix":"// before\ncatalog.write_to_parquet(&quotes, type_name, None, None, None, None, None)?;\n// after\nlet covered = catalog.get_directory_intervals(dir)?;\nlet disjoint = trim_to_uncovered(&quotes, covered)?; // drop/split overlapping rows\ncatalog.write_to_parquet(&disjoint, type_name, None, None, None, None, None)?;","handlingStrategy":"validation","validationCode":"let intervals = catalog.get_directory_intervals(&dir)?;\nlet new_range = (start_ts.as_u64(), end_ts.as_u64());\nassert!(!intervals.iter().any(|iv| overlaps(*iv, new_range)),\n    \"new write overlaps existing files; clear or trim first\");","typeGuard":null,"tryCatchPattern":"match catalog.write_to_parquet(&data, type_name, None, None, None, None, None) {\n    Err(e) if e.to_string().contains(\"non-disjoint intervals\") => {\n        clear_or_rename_directory(&dir)?;\n        catalog.write_to_parquet(&data, type_name, None, None, None, None, None)?;\n    }\n    other => other?,\n}","preventionTips":["Use a fresh catalog directory per recording session","Check existing intervals before each ingestion run","Delete stale files before re-ingesting the same range","Record (start,end) ranges in job metadata to detect overlap early"],"tags":["rust","persistence","parquet","intervals","duplicate-write"],"backgroundTag":"invalid-state-transition","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}