nautechsystems/nautilus_trader · error · anyhow::Error

Intervals are not disjoint after consolidating a directory

Error message

Intervals are not disjoint after consolidating a directory

What it means

After `consolidate_directory` merges/splits the parquet files in a directory, it recomputes every file's [start, end] ts_init interval and, when `ensure_contiguous_files` is enabled (the default), asserts the intervals are mutually disjoint. If overlapping intervals remain, the consolidation did not produce a clean, contiguous layout and the operation aborts to prevent ambiguous time-range lookups. This is an internal post-condition check on the catalog's file layout invariant.

Source

Thrown at crates/persistence/src/backend/catalog_operations.rs:341

                .iter()
                .map(|path| ObjectPath::from(path.as_str()))
                .collect();

            self.execute_async(async {
                combine_parquet_files_from_object_store(
                    self.object_store.clone(),
                    object_paths,
                    &ObjectPath::from(path),
                    Some(self.compression),
                    Some(self.max_row_group_size),
                    deduplicate,
                )
                .await
            })?;
        }

        if ensure_contiguous_files.unwrap_or(true) && !are_intervals_disjoint(&intervals) {
            anyhow::bail!("Intervals are not disjoint after consolidating a directory");
        }

        Ok(())
    }

    /// Consolidates all data files in the catalog by splitting them into fixed time periods.
    ///
    /// This method identifies all leaf directories in the catalog that contain parquet files
    /// and consolidates them by period. A leaf directory is one that contains files but no subdirectories.
    /// This is a convenience method that effectively calls `consolidate_data_by_period` for all data types
    /// and instrument IDs in the catalog.
    ///
    /// # Parameters
    ///
    /// - `period_nanos`: The period duration for consolidation in nanoseconds. Default is 1 day (86400000000000).
    ///   Examples: 3600000000000 (1 hour), 604800000000000 (7 days), 1800000000000 (30 minutes)
    /// - `start`: Optional start timestamp for the consolidation range. Only files with timestamps
    ///   greater than or equal to this value will be consolidated. If None, all files

View on GitHub (pinned to 18893faf8b)

Solutions

1. Stop all writers and re-run consolidation so no concurrent writes create overlapping files.
2. Set `ensure_contiguous_files: Some(false)` if overlapping files are acceptable for your workload, or pre-split consolidation into non-overlapping windows.
3. Inspect the instrument directory for duplicate/overlapping parquet files (same instrument, overlapping ts_init ranges) and delete or re-write them.
4. Run consolidation single-threaded / with an exclusive lock on the catalog directory.

Example fix

// before: consolidating while live writers are active
catalog.consolidate_catalog(None).await?;
// after: opt out of the contiguity requirement or ensure exclusive access
catalog.consolidate_catalog_url(None, Some(false)).await?; // ensure_contiguous_files = false

// Before consolidating, ensure no other process/thread writes to the catalog dir
// and that existing files are disjoint:
// fn files_disjoint(intervals: &[(u64, u64)]) -> bool {
//     let mut s = intervals.clone(); s.sort();
//     s.windows(2).all(|w| w[0].1 < w[1].0)
// }

match catalog.consolidate_catalog_url(None, None).await {
    Err(e) if e.to_string().contains("Intervals are not disjoint") => {
        // stop writers, inspect overlapping files, retry consolidation exclusively
    }
    Err(e) => return Err(e),
    Ok(()) => Ok(()),
}

When it happens

Trigger: Running `consolidate_catalog` or `consolidate_data` (which call `consolidate_directory`) where, after the consolidation pass, two or more files in the directory still have overlapping ts_init ranges — e.g. data written concurrently during consolidation, files outside the consolidated period that overlap, or duplicate/overlapping writes to the same instrument directory.

Common situations: A live strategy kept writing to the catalog while consolidation ran in another process/thread; pre-existing overlapping files from earlier crashes or manual copies; running two consolidation jobs at once; files whose periods overlap because an earlier consolidation used a different period setting.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• ts_init column contains null values
• Intervals are not contiguous. When ensure_contiguous_files=t
• Replacement hash {transaction_hash} conflicts with another i
• Included wrap transaction {tx_hash} has invalid block number
• WETH balance overflow for included transaction {tx_hash} at

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/ac888950392e838e. Report an issue: GitHub.