quickwit-oss/quickwit · error

window_start mismatch in

Error message

window_start mismatch in {}: expected {:?}, found {:?}

What it means

During a parquet merge, extract_and_validate_input_metadata reads the `window_start` metadata key from every input split's parquet file metadata and requires all inputs to agree. The first observed value becomes the consensus; any file whose window_start differs from that consensus aborts the merge. This prevents merging splits that cover different time windows, which would corrupt the merged output's window semantics.

Solutions

  1. Check each input file's parquet KV metadata `window_start` and only group files with identical values.
  2. Fix the merge/placement logic that selected the splits so it partitions by window_start.
  3. If the offending file was produced by a buggy writer, re-index or re-ingest it with correct window metadata.
  4. If merging across windows is intended, use an operation path that does not require window consensus.

Example fix

// before: mixing splits from different windows in one merge op
let op = ParquetMergeOperation { splits: vec![split_a_win1, split_b_win2], .. };
// after: group splits by window_start before merging
let mut groups: HashMap<Option<String>, Vec<Split>> = HashMap::new();
for s in splits { groups.entry(s.window_start.clone()).or_default().push(s); }
for group in groups.values() { merge(group.clone()).await?; }
Defensive patterns

Strategy: validation

Validate before calling

fn windows_consistent(files: &[PathBuf]) -> anyhow::Result<()> {
    let mut vals: HashSet<Option<String>> = HashSet::new();
    for f in files { vals.insert(read_parquet_kv(f, "window_start")); }
    anyhow::ensure!(vals.len() <= 1, "mixed window_start across merge inputs");
    Ok(())
}

Try / catch

match merge_sorted_parquet_files_impl(...).await {
    Err(e) if e.to_string().contains("window_start mismatch") => {
        // re-group splits by window and reschedule
    }
    other => other?,
}

Prevention

When it happens

Trigger: Calling merge_sorted_parquet_files_impl (or execute_merge_operation) with a set of parquet files whose embedded KV metadata `window_start` values are not all identical.

Common situations: Merging splits hand-picked from different time windows, mixing splits from an older writer that stamped windows differently, or a scheduling bug in the merge planner that groups non-contiguous windows.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/d8c0fd245360d3c8. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-parquet-engine/src/merge/mod.rs:407

                        "parsing sort schema from {}: '{}'",
                        path.display(),
                        file_sort_fields
                    )
                })?;
                consensus_sort_fields = Some(file_sort_fields.clone());
            }
        }

        // Window start: must be consistent.
        let file_window_start = find_kv(PARQUET_META_WINDOW_START)
            .map(|s| s.parse::<i64>())
            .transpose()
            .with_context(|| format!("parsing window_start from {}", path.display()))?;

        match &consensus_window_start {
            Some(expected) => {
                if file_window_start != *expected {
                    bail!(
                        "window_start mismatch in {}: expected {:?}, found {:?}",
                        path.display(),
                        expected,
                        file_window_start
                    );
                }
            }
            None => {
                consensus_window_start = Some(file_window_start);
            }
        }

        // Window duration: must be consistent.
        let file_window_duration = find_kv(PARQUET_META_WINDOW_DURATION)
            .map(|s| s.parse::<u32>())
            .transpose()
            .with_context(|| format!("parsing window_duration from {}", path.display()))?
            .unwrap_or(0);

View on GitHub (pinned to a39730c5cd)