quickwit-oss/quickwit · error

execute_merge_operation: sources.len() ({}) != op.splits.len

Error message

execute_merge_operation: sources.len() ({}) != op.splits.len() ({})

What it means

execute_merge_operation enforces that the number of byte sources passed in exactly matches the number of splits declared in the ParquetMergeOperation. Each split must have exactly one corresponding remote source; a mismatch means the caller built an inconsistent merge task, so it bails rather than silently merging fewer or more files than planned.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/mod.rs:522

/// range reads against `op.splits[i].parquet_file`. The caller (e.g.
/// the executor wrapper that lives outside this crate) is responsible
/// for materializing one [`RemoteByteSource`] per split based on its
/// storage backend (S3, local FS, etc.). Splits with names that
/// cannot be opened by the source surface as `LegacyAdapterError::Io`
/// or `ParquetReadError`.
///
/// Returns the merge engine's [`MergeOutputFile`]s. Conversion to
/// `ParquetSplitMetadata` for the metastore is the caller's
/// responsibility — use [`metadata_aggregation::merge_parquet_split_metadata`]
/// with `mixed_prefix_ok = op.target_prefix_len_override.is_some()`.
pub async fn execute_merge_operation(
    op: &policy::ParquetMergeOperation,
    sources: Vec<Arc<dyn RemoteByteSource>>,
    output_dir: &Path,
    config: &MergeConfig,
) -> Result<Vec<MergeOutputFile>> {
    if sources.len() != op.splits.len() {
        bail!(
            "execute_merge_operation: sources.len() ({}) != op.splits.len() ({})",
            sources.len(),
            op.splits.len(),
        );
    }

    let mut streams: Vec<Box<dyn ColumnPageStream>> = Vec::with_capacity(op.splits.len());
    for (split, source) in op.splits.iter().zip(sources) {
        let path = PathBuf::from(&split.parquet_file);
        let stream: Box<dyn ColumnPageStream> = match op.target_prefix_len_override {
            Some(target) if split.rg_partition_prefix_len < target => {
                // Promote this legacy input. The adapter re-encodes in
                // memory and presents itself as a prefix_len = target
                // single-RG stream to the merge engine.
                let adapter = LegacyInputAdapter::try_open(source, path, target)
                    .await
                    .with_context(|| {
                        format!(

View on GitHub (pinned to a39730c5cd)

Solutions

1. Ensure one Arc<dyn RemoteByteSource> is produced per split, in the same order, before calling execute_merge_operation.
2. Fix the download/fetch stage so failures abort the merge instead of dropping sources.
3. Build sources from op.splits itself rather than a separately maintained list.
4. If this appears in tests, update the test harness to derive sources from the op.

Example fix

// before: sources collected independently of op.splits
let sources = fetch_all(pending_files).await?;
execute_merge_operation(&op, sources, &out, &cfg).await?;
// after: derive sources from the op's splits
let sources: Vec<_> = op.splits.iter().map(|s| fetch_source(s)).collect::<Result<_>>()?;
execute_merge_operation(&op, sources, &out, &cfg).await?;

anyhow::ensure!(sources.len() == op.splits.len(),
    "sources ({}) must map 1:1 to op.splits ({})", sources.len(), op.splits.len());

match execute_merge_operation(&op, sources, &out, &cfg).await {
    Err(e) if e.to_string().contains("sources.len()") => {
        // refetch sources from op.splits and retry once
    }
    other => other?,
}

When it happens

Trigger: Calling execute_merge_operation (or the merge handler that wraps it) where sources.len() != op.splits.len() — e.g. a download step skipped or duplicated a source, or splits were added to the op after sources were materialized.

Common situations: A merge executor whose fetch stage filters out un-fetchable files without updating the operation; unit/integration harnesses (assert_engine_parity, tests) constructing the op and source list independently.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• attempted to merge splits with different doc mapping uid
• attempted to merge zero splits
• doc mapper must be present if there are delete tasks
• timestamp_secs must be UInt64 or Int64 for MC-3 check
• invalid arguments for `hash_mod`: expected 2 arguments, foun

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/fd5c367ea24e8237. Report an issue: GitHub.