quickwit-oss/quickwit · error

failed to build tokenizer `{}`: {:?}

Error message

failed to build tokenizer `{}`: {:?}

What it means

When building the index doc mapper, each custom tokenizer defined in `tokenizers` is instantiated via its text-analyzer configuration. If the analyzer options are invalid (bad token filters, unknown filter types, invalid options), the build fails and this error wraps the underlying cause together with the tokenizer name. It surfaces at index creation or config load time.

Source

Thrown at quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs:228

                    "duplicated custom tokenizer: `{}`",
                    tokenizer_config_entry.name
                );
            }
            if tokenizer_manager
                .get_tokenizer(&tokenizer_config_entry.name)
                .is_some()
            {
                bail!(
                    "custom tokenizer name `{}` should be different from built-in tokenizer's \
                     names",
                    tokenizer_config_entry.name
                );
            }
            let tokenizer = tokenizer_config_entry
                .config
                .text_analyzer()
                .map_err(|error| {
                    anyhow::anyhow!(
                        "failed to build tokenizer `{}`: {:?}",
                        tokenizer_config_entry.name,
                        error
                    )
                })?;
            let does_lowercasing = tokenizer_config_entry
                .config
                .filters
                .iter()
                .any(|filter| matches!(filter, crate::TokenFilterType::LowerCaser));
            tokenizer_manager.register(&tokenizer_config_entry.name, tokenizer, does_lowercasing);
            custom_tokenizer_names.insert(&tokenizer_config_entry.name);
        }
        validate_fields_tokenizers(&schema, &tokenizer_manager)?;

        // Resolve default search fields
        let mut default_search_field_names = Vec::new();
        for default_search_field_name in &builder.default_search_fields {

View on GitHub (pinned to a39730c5cd)

Solutions

1. Read the wrapped `{:?}` cause to identify which analyzer option failed.
2. Fix the offending token filter / normalizer entry in the `tokenizers` section of the index config.
3. Validate the config against the supported text analyzer options for your Quickwit version.
4. If a filter was removed in an upgrade, migrate the tokenizer definition to the supported equivalent.

Example fix

# before
tokenizers:
  my_tok:
    text_analyzer:
      token_filters:
        - type: stopwordz
# after
tokenizers:
  my_tok:
    text_analyzer:
      token_filters:
        - type: stop_words
          language: English

// Validate analyzer config before index creation by building the index config
// against a test index or using quickwit's config validation endpoint.
for tok in index_config.tokenizers.values() {
    tok.text_analyzer().expect("invalid tokenizer config");
}

match result {
    Err(e) if e.to_string().contains("failed to build tokenizer") => {
        // inspect wrapped cause, fix token_filters entry
    }
    other => other?,
}

When it happens

Trigger: Defining a `tokenizers:` entry in the index config whose `text_analyzer` options fail to build — e.g. unknown token filter, invalid lowercaser/normalizer option, malformed analyzer settings — and then creating the index or loading the doc mapper.

Common situations: Typo in a token filter name, copying an Elasticsearch analyzer definition incompatible with Quickwit's analyzer DSL, or a version change removing/renaming a filter option.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• index ID pattern `{pattern}` is invalid: patterns must not c
• index ID pattern `{pattern}` is invalid: an index ID must ha
• index config merge policy `max_merge_factor` must be superio
• VRL transforms are not supported for OTLP input formats
• duplicated custom tokenizer: `{}`

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/5be60c0dda5db358. Report an issue: GitHub.