quickwit-oss/quickwit · error

doc mapper must be present if there are delete tasks

Error message

doc mapper must be present if there are delete tasks

What it means

When merging splits that also apply delete tasks, the merged index must be opened with the index's doc mapper so delete queries can be translated into tantivy delete operations. If delete tasks exist but `doc_mapper_opt` is `None`, this invariant is violated and the merge fails with this error.

Source

Thrown at quickwit/quickwit-indexing/src/actors/merge_executor.rs:547

        let mut directory_stack: Vec<Box<dyn Directory>> = vec![
            output_directory.box_clone(),
            Box::new(shadowing_meta_json_directory),
        ];
        directory_stack.extend(split_directories);
        let union_directory = UnionDirectory::union_of(directory_stack);
        let union_index = open_index(
            union_directory,
            self.doc_mapper.tokenizer_manager().tantivy_manager(),
        )?;

        ctx.record_progress();
        let _protect_guard = ctx.protect_zone();

        let mut index_writer: IndexWriter = union_index.writer_with_num_threads(1, 15_000_000)?;
        let num_delete_tasks = delete_tasks.len();
        if num_delete_tasks > 0 {
            let doc_mapper = doc_mapper_opt
                .ok_or_else(|| anyhow!("doc mapper must be present if there are delete tasks"))?;
            for delete_task in delete_tasks {
                let delete_query = delete_task
                    .delete_query
                    .expect("A delete task must have a delete query.");
                let query_ast: QueryAst = serde_json::from_str(&delete_query.query_ast)
                    .context("invalid query_ast json")?;
                // We ignore the docmapper default fields when we consider delete query.
                // We reparse the query here defensively, but actually, it should already have been
                // done in the delete task rest handler.
                let parsed_query_ast = query_ast.parse_user_query(&[]).context("invalid query")?;
                debug!(
                    "Delete all documents matched by query `{:?}`",
                    parsed_query_ast
                );
                let (query, _) =
                    doc_mapper.query(union_index.schema(), parsed_query_ast, false, None)?;
                index_writer.delete_query(query)?;
            }

View on GitHub (pinned to a39730c5cd)

Solutions

1. Ensure the doc mapper is loaded from the metastore and passed into the merge operation whenever delete tasks exist.
2. Check whether the index metadata/doc mapping is intact in the metastore.
3. Retry the merge after confirming the index config is valid; investigate the caller that supplied `None`.
4. If it recurs with valid metadata, report as a Quickwit bug.

if !delete_tasks.is_empty() && doc_mapper_opt.is_none() {
    return Err("cannot run merge with delete tasks and no doc mapper");
}

match result {
    Err(e) if e.to_string().contains("doc mapper must be present") => {
        // reload index metadata and retry the merge
    }
    other => other?,
}

When it happens

Trigger: `merge_split_directories` is called from `process_merge` or `process_delete_and_merge` with a non-empty `delete_tasks` list while the merge plan/context carries no doc mapper (e.g. a merge-only path that never loaded the index doc mapping).

Common situations: Merges triggered on indexes whose doc mapping could not be loaded, delete tasks racing with an index config change, or internal code paths passing `None` for doc_mapper while deletes are pending.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• attempted to merge zero splits
• attempted to merge splits with different doc mapping uid
• execute_merge_operation: sources.len() ({}) != op.splits.len
• position of a Kinesis shard should never be EOF
• timestamp_secs must be UInt64 or Int64 for MC-3 check

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/9f6e4beade56ab1f. Report an issue: GitHub.