nautechsystems/nautilus_trader · error · anyhow::Error

options_chain CSV rows must be ordered by local_timestamp wh

Error message

options_chain CSV rows must be ordered by local_timestamp when thinning

What it means

When snapshot thinning is enabled for options-chain CSV conversion, rows are bucketed by `local_timestamp / interval_us` and only the first row of each bucket is kept. This requires the CSV to be sorted ascending by `local_timestamp`. If a later row belongs to an earlier bucket than the current one, the input is out of order and thinning would produce incorrect snapshots, so the conversion aborts.

Source

Thrown at crates/adapters/tardis/src/csv/convert.rs:129

            precision_by_instrument
                .entry(instrument_id)
                .or_insert_with(|| {
                    OptionsChainPrecision::new(config.price_precision, config.size_precision)
                })
                .update(&record, config.price_precision, config.size_precision);
            instrument_states
                .entry(instrument_id)
                .and_modify(|state| state.update_activation(record.local_timestamp))
                .or_insert_with(|| InstrumentBuildState::new(record.clone()));

            if let Some(interval) = config.snapshot_interval {
                let interval_us = u64::try_from(interval.as_micros())
                    .context("snapshot interval exceeds u64 microseconds")?;
                anyhow::ensure!(interval_us > 0, "snapshot interval must be positive");
                let bucket = record.local_timestamp / interval_us;

                if let Some(current_bucket) = current_bucket {
                    anyhow::ensure!(
                        bucket >= current_bucket,
                        "options_chain CSV rows must be ordered by local_timestamp when thinning"
                    );
                }

                if current_bucket.is_none_or(|current| bucket > current) {
                    flush_pending_records_before(
                        &catalog,
                        &mut pending_records,
                        &mut data_buffers,
                        &precision_by_instrument,
                        bucket,
                        config.extract_bbo_as_quotes,
                    )?;
                    current_bucket = Some(bucket);
                }
                pending_records
                    .entry((instrument_id, bucket))

View on GitHub (pinned to 18893faf8b)

Solutions

1. Sort the CSV by `local_timestamp` ascending before conversion, e.g. `sort -t, -k<col> file.csv` or a pandas `sort_values("local_timestamp")` pass.
2. Reorder the list of input files chronologically (the bucket state resets per file but ordering within each file still matters).
3. If the data cannot be ordered, disable thinning (`snapshot_interval: None`), which skips the ordering check.
4. Verify the export tool is sorting by `local_timestamp`, not `timestamp`.

Example fix

// before: convert unsorted CSV directly
convert_options_chain_csv(&path, &config)?;
// after: sort first
let mut df = CsvParser::from_path(&path)?;
df.sort(["local_timestamp"]);
df.to_csv(&path)?;
convert_options_chain_csv(&path, &config)?;

import csv

def ensure_sorted_by_local_timestamp(path: str, col: str = "local_timestamp") -> None:
    prev = None
    with open(path, newline="") as f:
        for row in csv.DictReader(f):
            ts = int(row[col])
            if prev is not None and ts < prev:
                raise ValueError(f"{path}: {col} out of order at ts={ts}")
            prev = ts

When it happens

Trigger: Calling `convert_options_chain_csv` with a `snapshot_interval` set while the CSV contains rows whose `local_timestamp` decreases relative to previously processed rows (within or across files in a multi-file run).

Common situations: Concatenating Tardis CSV exports in the wrong order; a provider export that is sorted by a different timestamp column (e.g. `timestamp` instead of `local_timestamp`); appending new data to an existing file without re-sorting.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• snapshot interval must be positive
• {e}
• {FAILED}: {e}
• {FAILED}: {e}
• {FAILED}: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/caa3056258fcbb13. Report an issue: GitHub.