quickwit-oss/quickwit · error

merge requires at least one input file

Error message

merge requires at least one input file

What it means

merge_sorted_parquet_files_impl bails immediately if `input_paths` is empty. A merge with zero input files has nothing to produce; the API treats that as a caller bug rather than returning an empty result, since callers are expected to only invoke the merge engine with at least one compaction candidate.

Solutions

1. Guard at the call site: skip the merge entirely if the candidate list is empty instead of invoking the engine.
2. Re-validate that all planned input paths still exist right before the merge and drop empty task sets.
3. Check for concurrent merge workers double-claiming tasks and adding appropriate task claiming/locking.
4. In tests, assert candidate lists are non-empty before constructing merge tasks.

Example fix

// before
run_merge_task(task)?; // task.input_paths may be empty
// after
if task.input_paths.is_empty() {
    info!(task_id = %task.id, "no inputs left, skipping merge");
    return Ok(());
}
run_merge_task(task)?;

if input_paths.is_empty() {
    // skip merge entirely
    return Ok(());
}

fn non_empty<'a, T>(paths: &'a [T]) -> Option<&'a [T]> {
    (!paths.is_empty()).then_some(paths)
}

match merge_sorted_parquet_files(&paths, out_dir, &config) {
    Err(e) if e.to_string().contains("at least one input file") => {
        info!("merge task became empty (inputs deleted?); skipping");
        Ok(())
    }
    other => other,
}

When it happens

Trigger: Calling merge_sorted_parquet_files(input_paths, output_dir, config) with an empty slice — e.g. a merge policy that built a task before checking candidate count, or a filter that removed all paths after task creation.

Common situations: Race where candidate splits were deleted (retention/expiry) between planning and execution; a merge task persisted in the metastore whose inputs were already merged by another worker; unit tests calling the merge function directly with no fixtures.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• num_outputs must be at least 1
• concatenate type must have at least one sub-field
• fast field is not allowed for array
• field name ` ` is reserved. the following fields are…
• field name is empty

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/a1c471d0d9899a26. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-parquet-engine/src/merge/mod.rs:178

/// the multi-RecordBatch concatenation path.
#[cfg(test)]
pub(crate) fn merge_sorted_parquet_files_with_read_batch_size(
    input_paths: &[PathBuf],
    output_dir: &Path,
    config: &MergeConfig,
    read_batch_size: usize,
) -> Result<Vec<MergeOutputFile>> {
    merge_sorted_parquet_files_impl(input_paths, output_dir, config, Some(read_batch_size))
}

fn merge_sorted_parquet_files_impl(
    input_paths: &[PathBuf],
    output_dir: &Path,
    config: &MergeConfig,
    read_batch_size: Option<usize>,
) -> Result<Vec<MergeOutputFile>> {
    if input_paths.is_empty() {
        bail!("merge requires at least one input file");
    }
    if config.num_outputs == 0 {
        bail!("num_outputs must be at least 1");
    }

    // Step 0: Read and validate metadata from all input files.
    // Sort schema, window, and merge ops are derived from the files themselves.
    let input_meta = extract_and_validate_input_metadata(input_paths)?;

    info!(
        num_inputs = input_paths.len(),
        num_outputs = config.num_outputs,
        sort_fields = %input_meta.sort_fields,
        "starting sorted parquet merge"
    );

    // Step 1: Read all input files into RecordBatches.
    let inputs = read_inputs(input_paths, read_batch_size)?;

View on GitHub (pinned to a39730c5cd)