quickwit-oss/quickwit · error

window_start mismatch in {}: expected {:?}, found {:?}

Error message

window_start mismatch in {}: expected {:?}, found {:?}

What it means

During a parquet merge, extract_and_validate_input_metadata reads the `window_start` metadata key from every input split's parquet file metadata and requires all inputs to agree. The first observed value becomes the consensus; any file whose window_start differs from that consensus aborts the merge. This prevents merging splits that cover different time windows, which would corrupt the merged output's window semantics.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/mod.rs:407

                        "parsing sort schema from {}: '{}'",
                        path.display(),
                        file_sort_fields
                    )
                })?;
                consensus_sort_fields = Some(file_sort_fields.clone());
            }
        }

        // Window start: must be consistent.
        let file_window_start = find_kv(PARQUET_META_WINDOW_START)
            .map(|s| s.parse::<i64>())
            .transpose()
            .with_context(|| format!("parsing window_start from {}", path.display()))?;

        match &consensus_window_start {
            Some(expected) => {
                if file_window_start != *expected {
                    bail!(
                        "window_start mismatch in {}: expected {:?}, found {:?}",
                        path.display(),
                        expected,
                        file_window_start
                    );
                }
            }
            None => {
                consensus_window_start = Some(file_window_start);
            }
        }

        // Window duration: must be consistent.
        let file_window_duration = find_kv(PARQUET_META_WINDOW_DURATION)
            .map(|s| s.parse::<u32>())
            .transpose()
            .with_context(|| format!("parsing window_duration from {}", path.display()))?
            .unwrap_or(0);

View on GitHub (pinned to a39730c5cd)

Solutions

  1. Check each input file's parquet KV metadata `window_start` and only group files with identical values.
  2. Fix the merge/placement logic that selected the splits so it partitions by window_start.
  3. If the offending file was produced by a buggy writer, re-index or re-ingest it with correct window metadata.
  4. If merging across windows is intended, use an operation path that does not require window consensus.

Example fix

// before: mixing splits from different windows in one merge op
let op = ParquetMergeOperation { splits: vec![split_a_win1, split_b_win2], .. };
// after: group splits by window_start before merging
let mut groups: HashMap<Option<String>, Vec<Split>> = HashMap::new();
for s in splits { groups.entry(s.window_start.clone()).or_default().push(s); }
for group in groups.values() { merge(group.clone()).await?; }
Defensive patterns

Strategy: validation

Validate before calling

fn windows_consistent(files: &[PathBuf]) -> anyhow::Result<()> {
    let mut vals: HashSet<Option<String>> = HashSet::new();
    for f in files { vals.insert(read_parquet_kv(f, "window_start")); }
    anyhow::ensure!(vals.len() <= 1, "mixed window_start across merge inputs");
    Ok(())
}

Try / catch

match merge_sorted_parquet_files_impl(...).await {
    Err(e) if e.to_string().contains("window_start mismatch") => {
        // re-group splits by window and reschedule
    }
    other => other?,
}

Prevention

When it happens

Trigger: Calling merge_sorted_parquet_files_impl (or execute_merge_operation) with a set of parquet files whose embedded KV metadata `window_start` values are not all identical.

Common situations: Merging splits hand-picked from different time windows, mixing splits from an older writer that stamped windows differently, or a scheduling bug in the merge planner that groups non-contiguous windows.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/d8c0fd245360d3c8. Report an issue: GitHub.