{"record":{"id":"a35bbf14add4d724","repo":"quickwit-oss/quickwit","slug":"unknown-tokenizer-for-field","errorCode":null,"errorMessage":"unknown tokenizer `{}` for field `{}`","messagePattern":"unknown tokenizer `(.+?)` for field `(.+?)`","errorType":"validation","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs","lineNumber":374,"sourceCode":"/// Checks that a given text/json field name has a registered tokenizer.\nfn validate_fields_tokenizers(\n    schema: &Schema,\n    tokenizer_manager: &TokenizerManager,\n) -> Result<(), anyhow::Error> {\n    for (_, field_entry) in schema.fields() {\n        let tokenizer_name_opt = match field_entry.field_type() {\n            FieldType::Str(options) => options\n                .get_indexing_options()\n                .map(|text_options: &tantivy::schema::TextFieldIndexing| text_options.tokenizer()),\n            FieldType::JsonObject(options) => options\n                .get_text_indexing_options()\n                .map(|text_options: &tantivy::schema::TextFieldIndexing| text_options.tokenizer()),\n            _ => None,\n        };\n        if let Some(tokenizer_name) = tokenizer_name_opt\n            && tokenizer_manager.get_tokenizer(tokenizer_name).is_none()\n        {\n            bail!(\n                \"unknown tokenizer `{}` for field `{}`\",\n                tokenizer_name,\n                field_entry.name()\n            );\n        }\n    }\n    Ok(())\n}\n\nimpl std::fmt::Debug for DocMapper {\n    fn fmt(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {\n        formatter\n            .debug_struct(\"DocMapper\")\n            .field(\"store_source\", &self.source_field.is_some())\n            .field(\n                \"default_search_field_names\",\n                &self.default_search_field_names,\n            )","sourceCodeStart":356,"sourceCodeEnd":392,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs#L356-L392","documentation":"Quickwit validates at index-creation time that every text field referencing a named tokenizer actually has that tokenizer registered in the tokenizer manager (built-in tokenizers like 'raw', 'default', 'en_stem' or user-defined ones from the config). If the field's indexing options name a tokenizer the manager does not know, mapping construction fails with this message. It is a configuration-time guard so bad mappings never reach indexing.","triggerScenarios":"Calling IndexConfig / DocMapper::try_from with a field mapping whose text options set `tokenizer` (or `record: basic|with_freqs_and_positions` implying a tokenizer) to a name not defined in the search settings and not a built-in, e.g. a typo like `tokenizer: en_stemm`.","commonSituations":"Typo in tokenizer name; referencing a custom tokenizer defined in another node's config but missing locally; renaming/removing a custom tokenizer in quickwit config while old index mappings still reference it; copying mappings between clusters with different tokenizer setups.","solutions":["Fix the `tokenizer` value in the field mapping to a registered name (built-ins: default, raw, en_stem, non_stem plus any custom tokenizers declared in `search_settings` / tokenizers config).","Add the missing custom tokenizer definition to the quickwit config's tokenizer section so the tokenizer manager knows it.","Check for typos by listing available tokenizers in your config and comparing names exactly (names are case-sensitive)."],"exampleFix":"// before\n{name: \"body\", type: \"text\", tokenizer: \"en_stemm\"}\n// after\n{name: \"body\", type: \"text\", tokenizer: \"en_stem\"}","handlingStrategy":"validation","validationCode":"// before submitting the index config\nlet available: &[&str] = &[\"default\", \"raw\", \"en_stem\", \"non_stem\", /* + custom names from config */];\nfn tokenizer_ok(field: &serde_json::Value, available: &[&str]) -> bool {\n    match field.get(\"tokenizer\").and_then(|t| t.as_str()) {\n        None => true,\n        Some(name) => available.contains(&name),\n    }\n}","typeGuard":null,"tryCatchPattern":"// match on config create/update error and surface tokenizer name to the user\nmatch create_index_result {\n    Err(e) if e.to_string().contains(\"unknown tokenizer\") => {\n        eprintln!(\"Fix the tokenizer name in your mapping: {e}\");\n    }\n    other => other?,\n}","preventionTips":["Keep a canonical list of tokenizer names in your pipeline config and validate mappings against it.","Never hand-type tokenizer names; reference constants from a shared config.","Ensure all nodes in the cluster share the same tokenizer definitions."],"tags":["search","tokenizer","config","rust"],"backgroundTag":"invalid-config-value","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}