quickwit-oss/quickwit · error · anyhow::Error

custom tokenizer name `{}` should be different from built-in

Error message

custom tokenizer name `{}` should be different from built-in tokenizer's names

What it means

Custom tokenizer names are validated against Quickwit's built-in tokenizer set (default, raw, en_stem, etc.) before registration. Shadowing a built-in name would create ambiguity about which tokenizer is used, so it is rejected.

Source

Thrown at quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs:218

        } else {
            None
        };
        let schema = schema_builder.build();

        let tokenizer_manager = create_default_quickwit_tokenizer_manager();
        let mut custom_tokenizer_names = HashSet::new();
        for tokenizer_config_entry in &doc_mapping.tokenizers {
            if custom_tokenizer_names.contains(&tokenizer_config_entry.name) {
                bail!(
                    "duplicated custom tokenizer: `{}`",
                    tokenizer_config_entry.name
                );
            }
            if tokenizer_manager
                .get_tokenizer(&tokenizer_config_entry.name)
                .is_some()
            {
                bail!(
                    "custom tokenizer name `{}` should be different from built-in tokenizer's \
                     names",
                    tokenizer_config_entry.name
                );
            }
            let tokenizer = tokenizer_config_entry
                .config
                .text_analyzer()
                .map_err(|error| {
                    anyhow::anyhow!(
                        "failed to build tokenizer `{}`: {:?}",
                        tokenizer_config_entry.name,
                        error
                    )
                })?;
            let does_lowercasing = tokenizer_config_entry
                .config
                .filters

View on GitHub (pinned to a39730c5cd)

Solutions

Rename the custom tokenizer to something not colliding with built-in names (e.g. prefix it: `my_default`).
If the built-in behavior is wanted, remove the custom entry and reference the built-in tokenizer directly in field options.

Example fix

# before
tokenizers:
  - name: raw
    type: ngram

# after
tokenizers:
  - name: custom_raw_ngram
    type: ngram

Defensive patterns

Strategy: validation

Validate before calling

const BUILTIN_TOKENIZERS: [&str; 7] = ["raw", "default", "en_stem", "whitespace", "liju", "chinese_compatible", "multilang"];
fn check_builtin_collision(cfg: &serde_yaml::Value) -> Result<(), String> {
    if let Some(toks) = cfg["doc_mapping"]["tokenizers"].as_sequence() {
        for t in toks {
            let name = t["name"].as_str().unwrap_or_default();
            if BUILTIN_TOKENIZERS.contains(&name) {
                return Err(format!("custom tokenizer name `{name}` collides with a built-in"));
            }
        }
    }
    Ok(())
}

Try / catch

if let Err(e) = QuickwitDocMapper::try_from(&index_config) {
    if e.to_string().contains("built-in tokenizer") {
        eprintln!("Rename the custom tokenizer to avoid a built-in name: {e}");
    }
    return Err(e);
}

Prevention

Never name custom tokenizers after built-ins (`raw`, `default`, language tokenizers).
Prefix custom tokenizer names with a project or team prefix.
Check the tokenizer manager docs for the built-in name list before naming.

When it happens

Trigger: `QuickwitDocMapper::try_from` where an entry in `doc_mapping.tokenizers` has a `name` matching a tokenizer already registered in the default Quickwit tokenizer manager.

Common situations: Naming a custom tokenizer `default` or `raw` assuming custom names take a separate namespace; porting configs from systems where such names are allowed.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/4a89c2408e325c08. Report an issue: GitHub.