quickwit-oss/quickwit · error

merge requires at least one input file

Error message

merge requires at least one input file

What it means

merge_sorted_parquet_files_impl bails immediately if `input_paths` is empty. A merge with zero input files has nothing to produce; the API treats that as a caller bug rather than returning an empty result, since callers are expected to only invoke the merge engine with at least one compaction candidate.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/mod.rs:178

/// the multi-RecordBatch concatenation path.
#[cfg(test)]
pub(crate) fn merge_sorted_parquet_files_with_read_batch_size(
    input_paths: &[PathBuf],
    output_dir: &Path,
    config: &MergeConfig,
    read_batch_size: usize,
) -> Result<Vec<MergeOutputFile>> {
    merge_sorted_parquet_files_impl(input_paths, output_dir, config, Some(read_batch_size))
}

fn merge_sorted_parquet_files_impl(
    input_paths: &[PathBuf],
    output_dir: &Path,
    config: &MergeConfig,
    read_batch_size: Option<usize>,
) -> Result<Vec<MergeOutputFile>> {
    if input_paths.is_empty() {
        bail!("merge requires at least one input file");
    }
    if config.num_outputs == 0 {
        bail!("num_outputs must be at least 1");
    }

    // Step 0: Read and validate metadata from all input files.
    // Sort schema, window, and merge ops are derived from the files themselves.
    let input_meta = extract_and_validate_input_metadata(input_paths)?;

    info!(
        num_inputs = input_paths.len(),
        num_outputs = config.num_outputs,
        sort_fields = %input_meta.sort_fields,
        "starting sorted parquet merge"
    );

    // Step 1: Read all input files into RecordBatches.
    let inputs = read_inputs(input_paths, read_batch_size)?;

View on GitHub (pinned to a39730c5cd)

Solutions

1. Guard at the call site: skip the merge entirely if the candidate list is empty instead of invoking the engine.
2. Re-validate that all planned input paths still exist right before the merge and drop empty task sets.
3. Check for concurrent merge workers double-claiming tasks and adding appropriate task claiming/locking.
4. In tests, assert candidate lists are non-empty before constructing merge tasks.

Example fix

// before
run_merge_task(task)?; // task.input_paths may be empty
// after
if task.input_paths.is_empty() {
    info!(task_id = %task.id, "no inputs left, skipping merge");
    return Ok(());
}
run_merge_task(task)?;

if input_paths.is_empty() {
    // skip merge entirely
    return Ok(());
}

fn non_empty<'a, T>(paths: &'a [T]) -> Option<&'a [T]> {
    (!paths.is_empty()).then_some(paths)
}

match merge_sorted_parquet_files(&paths, out_dir, &config) {
    Err(e) if e.to_string().contains("at least one input file") => {
        info!("merge task became empty (inputs deleted?); skipping");
        Ok(())
    }
    other => other,
}

When it happens

Trigger: Calling merge_sorted_parquet_files(input_paths, output_dir, config) with an empty slice — e.g. a merge policy that built a task before checking candidate count, or a filter that removed all paths after task creation.

Common situations: Race where candidate splits were deleted (retention/expiry) between planning and execution; a merge task persisted in the metastore whose inputs were already merged by another worker; unit tests calling the merge function directly with no fixtures.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• num_outputs must be at least 1
• object type must have at least one field mapping
• concatenate type must have at least one sub-field
• fast field is not allowed for array<bytes>
• field name `{field_mapping_name}` is reserved. the following

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/a1c471d0d9899a26. Report an issue: GitHub.