nautechsystems/nautilus_trader · error · anyhow::Error

Writing file {filename} with interval ({start_ts}, {end_ts})

Error message

Writing file {filename} with interval ({start_ts}, {end_ts}) would create non-disjoint intervals. Existing intervals: {current_intervals:?}

What it means

`write_to_parquet` maintains non-overlapping (disjoint) time intervals per data directory. Before writing, it collects existing file intervals in the target directory and rejects the write if the new file's `(ts_start, ts_end)` would overlap any of them, because overlapping files make reads ambiguous.

Source

Thrown at crates/persistence/src/backend/catalog.rs:646

        let file_exists = self.execute_async(async {
            let exists: bool = self.object_store.head(&object_path).await.is_ok();
            Ok(exists)
        })?;

        if file_exists {
            log::info!("File {} already exists, skipping write", path.display());
            return Ok(path);
        }

        if !skip_disjoint_check.unwrap_or(false) {
            let current_intervals = self.get_directory_intervals(&directory)?;
            let new_interval = (start_ts.as_u64(), end_ts.as_u64());
            let mut new_intervals = current_intervals.clone();
            new_intervals.push(new_interval);

            if !are_intervals_disjoint(&new_intervals) {
                anyhow::bail!(
                    "Writing file {filename} with interval ({start_ts}, {end_ts}) would create \
                    non-disjoint intervals. Existing intervals: {current_intervals:?}"
                );
            }
        }

        log::info!(
            "Writing {} batches of {type_name} data to {}",
            batches.len(),
            path.display(),
        );

        self.execute_async(async {
            write_batches_to_object_store(
                &batches,
                self.object_store.clone(),
                &object_path,
                Some(self.compression),

View on GitHub (pinned to 18893faf8b)

Solutions

1. Delete or move the existing overlapping files in the catalog directory before rewriting.
2. Use a fresh catalog directory or session name for each recording run.
3. Write only the time range not already covered, or use `extend_file_name` to append to the existing file instead.
4. Check existing intervals first via the catalog's directory interval listing and trim the new batch accordingly.

Example fix

// before
catalog.write_to_parquet(&quotes, type_name, None, None, None, None, None)?;
// after
let covered = catalog.get_directory_intervals(dir)?;
let disjoint = trim_to_uncovered(&quotes, covered)?; // drop/split overlapping rows
catalog.write_to_parquet(&disjoint, type_name, None, None, None, None, None)?;

let intervals = catalog.get_directory_intervals(&dir)?;
let new_range = (start_ts.as_u64(), end_ts.as_u64());
assert!(!intervals.iter().any(|iv| overlaps(*iv, new_range)),
    "new write overlaps existing files; clear or trim first");

match catalog.write_to_parquet(&data, type_name, None, None, None, None, None) {
    Err(e) if e.to_string().contains("non-disjoint intervals") => {
        clear_or_rename_directory(&dir)?;
        catalog.write_to_parquet(&data, type_name, None, None, None, None, None)?;
    }
    other => other?,
}

When it happens

Trigger: Writing the same data type into the same catalog directory twice with overlapping time ranges — e.g. re-running an ingestion job, re-recording a session into the same catalog, or appending data whose timestamps fall inside an already-written file's range.

Common situations: Re-running a recording script without clearing the catalog; ingesting overlapping historical chunks; a live recorder restarted while old files remain in the directory.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Intervals are not disjoint after extending a file
• Unsupported Data::Defi variant for catalog writes
• Unsupported Data variant for catalog writes
• Cannot write {type_name} data with mixed identities: element
• {type_name} timestamps must be in ascending order

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/8fc37f811d50611a. Report an issue: GitHub.