nautechsystems/nautilus_trader · error · anyhow::Error

Intervals are not disjoint after extending a file

Error message

Intervals are not disjoint after extending a file

What it means

`extend_file_name` merges a source file into a target file within a directory and then re-validates that the directory's file intervals are still disjoint. If the extended file's new interval overlaps a neighbor's, the operation is rolled back semantically and the error is raised, since the post-condition invariant was violated.

Source

Thrown at crates/persistence/src/backend/catalog.rs:1288

        let start = start.as_u64();
        let end = end.as_u64();

        for interval in intervals {
            if interval.0 == end + 1 {
                // Extend backwards: new file covers [start, interval.1]
                self.rename_parquet_file(&directory, interval.0, interval.1, start, interval.1)?;
                break;
            } else if interval.1 == start - 1 {
                // Extend forwards: new file covers [interval.0, end]
                self.rename_parquet_file(&directory, interval.0, interval.1, interval.0, end)?;
                break;
            }
        }

        let intervals = self.get_directory_intervals(&directory)?;

        if !are_intervals_disjoint(&intervals) {
            anyhow::bail!("Intervals are not disjoint after extending a file");
        }

        Ok(())
    }

    /// Lists all Parquet files in a specified directory.
    ///
    /// This method scans a directory and returns the full paths of all files with the `.parquet`
    /// extension. It works with both local filesystems and remote object stores, making it
    /// suitable for various storage backends.
    ///
    /// # Parameters
    ///
    /// - `directory`: The directory path to scan for Parquet files.
    ///
    /// # Returns
    ///
    /// Returns a vector of full file paths (as strings) for all Parquet files found in the directory.

View on GitHub (pinned to 18893faf8b)

Solutions

1. Choose a target file whose interval fully contains the source file's interval.
2. Check directory intervals before extending and pick the correct neighbor.
3. Split the source file and extend each part into its appropriate target.
4. If overlaps are intentional, restructure the directory (rewrite all files) rather than extending.

Example fix

// before
catalog.extend_file_name(dir, source_file, target_file)?;
// after
let intervals = catalog.get_directory_intervals(dir)?;
if contains(intervals, source_range, target_file) {
    catalog.extend_file_name(dir, source_file, target_file)?;
} else {
    // pick the file whose interval contains source_range, or split the source
}

let intervals = catalog.get_directory_intervals(&dir)?;
// only extend when the source range is contained by the target's interval
assert!(intervals.iter().any(|iv| contains(iv, source_range)),
    "source range not contained by any existing interval");

match catalog.extend_file_name(&dir, &source, &target) {
    Err(e) if e.to_string().contains("not disjoint after extending") => {
        // wrong target: pick the file whose interval contains the source range
        let target2 = pick_containing_file(&dir, &source)?;
        catalog.extend_file_name(&dir, &source, &target2)?;
    }
    other => other?,
}

When it happens

Trigger: Calling `extend_file_name` with a source file whose time range spills outside the target file's interval into an adjacent file's range — e.g. extending a file with data that belongs to the next segment.

Common situations: Manual file consolidation/cleanup scripts picking the wrong target file; merging segments after deleting an intermediate file; automated compaction that mis-assigned boundaries.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Writing file {filename} with interval ({start_ts}, {end_ts})
• Unsupported Data::Defi variant for catalog writes
• Unsupported Data variant for catalog writes
• Cannot write {type_name} data with mixed identities: element
• {type_name} timestamps must be in ascending order

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/5438539f3a9fba06. Report an issue: GitHub.