quickwit-oss/quickwit · error · anyhow::Error

duplicated custom tokenizer: `{}`

Error message

duplicated custom tokenizer: `{}`

What it means

While building the tokenizer manager in try_from, each custom tokenizer defined in `doc_mapping.tokenizers` is checked against a HashSet of already-seen names. Defining two custom tokenizers with the same name is ambiguous (which config wins?) so construction fails immediately.

Source

Thrown at quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs:209

            concatenate_dynamic_fields,
        } = build_mapping_tree(&doc_mapping.field_mappings, &mut schema_builder)?;
        if !concatenate_dynamic_fields.is_empty() && dynamic_field.is_none() {
            bail!("concatenate field has `include_dynamic_fields` set, but index isn't dynamic");
        }
        let timestamp_field_path = if let Some(timestamp_field_name) = &doc_mapping.timestamp_field
        {
            validate_timestamp_field(timestamp_field_name, &field_mappings)?;
            Some(build_field_path_from_str(timestamp_field_name))
        } else {
            None
        };
        let schema = schema_builder.build();

        let tokenizer_manager = create_default_quickwit_tokenizer_manager();
        let mut custom_tokenizer_names = HashSet::new();
        for tokenizer_config_entry in &doc_mapping.tokenizers {
            if custom_tokenizer_names.contains(&tokenizer_config_entry.name) {
                bail!(
                    "duplicated custom tokenizer: `{}`",
                    tokenizer_config_entry.name
                );
            }
            if tokenizer_manager
                .get_tokenizer(&tokenizer_config_entry.name)
                .is_some()
            {
                bail!(
                    "custom tokenizer name `{}` should be different from built-in tokenizer's \
                     names",
                    tokenizer_config_entry.name
                );
            }
            let tokenizer = tokenizer_config_entry
                .config
                .text_analyzer()
                .map_err(|error| {

View on GitHub (pinned to a39730c5cd)

Solutions

1. Rename one of the duplicate custom tokenizers to a unique name.
2. Delete the redundant tokenizer entry.
3. Update the field(s) referencing the removed tokenizer to use the surviving one.

Example fix

# before
tokenizers:
  - name: my_tok
    type: ngram
  - name: my_tok
    type: raw

# after
tokenizers:
  - name: my_ngram_tok
    type: ngram
  - name: my_raw_tok
    type: raw

fn check_dup_tokenizers(cfg: &serde_yaml::Value) -> Result<(), String> {
    let mut seen = std::collections::HashSet::new();
    if let Some(toks) = cfg["doc_mapping"]["tokenizers"].as_sequence() {
        for t in toks {
            let name = t["name"].as_str().unwrap_or_default();
            if !seen.insert(name) {
                return Err(format!("duplicated custom tokenizer: {name}"));
            }
        }
    }
    Ok(())
}

match QuickwitDocMapper::try_from(&index_config) {
    Ok(m) => m,
    Err(e) if e.to_string().starts_with("duplicated custom tokenizer") => {
        eprintln!("Fix doc_mapping.tokenizers: {e}");
        return Err(e);
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: `QuickwitDocMapper::try_from(index_config)` with two entries in `doc_mapping.tokenizers` sharing the same `name` value.

Common situations: Merging two index config YAML files or appending tokenizer definitions to an existing config without noticing the name already exists.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• custom tokenizer name `{}` should be different from built-in
• duplicated default search field: `{}`
• tags collection is only allowed on text fields with the `raw
• unknown tokenizer `{}` for field `{}`
• failed to build tokenizer `{}`: {:?}

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/d224de3e3488cd23. Report an issue: GitHub.