quickwit-oss/quickwit · error · anyhow::Error

unknown tokenizer `{}` for field `{}`

Error message

unknown tokenizer `{}` for field `{}`

What it means

Quickwit validates at index-creation time that every text field referencing a named tokenizer actually has that tokenizer registered in the tokenizer manager (built-in tokenizers like 'raw', 'default', 'en_stem' or user-defined ones from the config). If the field's indexing options name a tokenizer the manager does not know, mapping construction fails with this message. It is a configuration-time guard so bad mappings never reach indexing.

Source

Thrown at quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs:374

/// Checks that a given text/json field name has a registered tokenizer.
fn validate_fields_tokenizers(
    schema: &Schema,
    tokenizer_manager: &TokenizerManager,
) -> Result<(), anyhow::Error> {
    for (_, field_entry) in schema.fields() {
        let tokenizer_name_opt = match field_entry.field_type() {
            FieldType::Str(options) => options
                .get_indexing_options()
                .map(|text_options: &tantivy::schema::TextFieldIndexing| text_options.tokenizer()),
            FieldType::JsonObject(options) => options
                .get_text_indexing_options()
                .map(|text_options: &tantivy::schema::TextFieldIndexing| text_options.tokenizer()),
            _ => None,
        };
        if let Some(tokenizer_name) = tokenizer_name_opt
            && tokenizer_manager.get_tokenizer(tokenizer_name).is_none()
        {
            bail!(
                "unknown tokenizer `{}` for field `{}`",
                tokenizer_name,
                field_entry.name()
            );
        }
    }
    Ok(())
}

impl std::fmt::Debug for DocMapper {
    fn fmt(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
        formatter
            .debug_struct("DocMapper")
            .field("store_source", &self.source_field.is_some())
            .field(
                "default_search_field_names",
                &self.default_search_field_names,
            )

View on GitHub (pinned to a39730c5cd)

Solutions

1. Fix the `tokenizer` value in the field mapping to a registered name (built-ins: default, raw, en_stem, non_stem plus any custom tokenizers declared in `search_settings` / tokenizers config).
2. Add the missing custom tokenizer definition to the quickwit config's tokenizer section so the tokenizer manager knows it.
3. Check for typos by listing available tokenizers in your config and comparing names exactly (names are case-sensitive).

Example fix

// before
{name: "body", type: "text", tokenizer: "en_stemm"}
// after
{name: "body", type: "text", tokenizer: "en_stem"}

// before submitting the index config
let available: &[&str] = &["default", "raw", "en_stem", "non_stem", /* + custom names from config */];
fn tokenizer_ok(field: &serde_json::Value, available: &[&str]) -> bool {
    match field.get("tokenizer").and_then(|t| t.as_str()) {
        None => true,
        Some(name) => available.contains(&name),
    }
}

// match on config create/update error and surface tokenizer name to the user
match create_index_result {
    Err(e) if e.to_string().contains("unknown tokenizer") => {
        eprintln!("Fix the tokenizer name in your mapping: {e}");
    }
    other => other?,
}

When it happens

Trigger: Calling IndexConfig / DocMapper::try_from with a field mapping whose text options set `tokenizer` (or `record: basic|with_freqs_and_positions` implying a tokenizer) to a name not defined in the search settings and not a built-in, e.g. a typo like `tokenizer: en_stemm`.

Common situations: Typo in tokenizer name; referencing a custom tokenizer defined in another node's config but missing locally; renaming/removing a custom tokenizer in quickwit config while old index mappings still reference it; copying mappings between clusters with different tokenizer setups.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• the `compactor` service can only be enabled when `enable_sta
• failed to parse service `{service_str}`. supported services
• failed to compile VRL script: {formatter}
• stdin can only be used as source through the CLI command `qu
• Quickwit currently supports multiple pipelines only for GCP

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/a35bbf14add4d724. Report an issue: GitHub.