{"record":{"id":"5be60c0dda5db358","repo":"quickwit-oss/quickwit","slug":"failed-to-build-tokenizer","errorCode":null,"errorMessage":"failed to build tokenizer `{}`: {:?}","messagePattern":"failed to build tokenizer `(.+?)`: (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs","lineNumber":228,"sourceCode":"                    \"duplicated custom tokenizer: `{}`\",\n                    tokenizer_config_entry.name\n                );\n            }\n            if tokenizer_manager\n                .get_tokenizer(&tokenizer_config_entry.name)\n                .is_some()\n            {\n                bail!(\n                    \"custom tokenizer name `{}` should be different from built-in tokenizer's \\\n                     names\",\n                    tokenizer_config_entry.name\n                );\n            }\n            let tokenizer = tokenizer_config_entry\n                .config\n                .text_analyzer()\n                .map_err(|error| {\n                    anyhow::anyhow!(\n                        \"failed to build tokenizer `{}`: {:?}\",\n                        tokenizer_config_entry.name,\n                        error\n                    )\n                })?;\n            let does_lowercasing = tokenizer_config_entry\n                .config\n                .filters\n                .iter()\n                .any(|filter| matches!(filter, crate::TokenFilterType::LowerCaser));\n            tokenizer_manager.register(&tokenizer_config_entry.name, tokenizer, does_lowercasing);\n            custom_tokenizer_names.insert(&tokenizer_config_entry.name);\n        }\n        validate_fields_tokenizers(&schema, &tokenizer_manager)?;\n\n        // Resolve default search fields\n        let mut default_search_field_names = Vec::new();\n        for default_search_field_name in &builder.default_search_fields {","sourceCodeStart":210,"sourceCodeEnd":246,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs#L210-L246","documentation":"When building the index doc mapper, each custom tokenizer defined in `tokenizers` is instantiated via its text-analyzer configuration. If the analyzer options are invalid (bad token filters, unknown filter types, invalid options), the build fails and this error wraps the underlying cause together with the tokenizer name. It surfaces at index creation or config load time.","triggerScenarios":"Defining a `tokenizers:` entry in the index config whose `text_analyzer` options fail to build — e.g. unknown token filter, invalid lowercaser/normalizer option, malformed analyzer settings — and then creating the index or loading the doc mapper.","commonSituations":"Typo in a token filter name, copying an Elasticsearch analyzer definition incompatible with Quickwit's analyzer DSL, or a version change removing/renaming a filter option.","solutions":["Read the wrapped `{:?}` cause to identify which analyzer option failed.","Fix the offending token filter / normalizer entry in the `tokenizers` section of the index config.","Validate the config against the supported text analyzer options for your Quickwit version.","If a filter was removed in an upgrade, migrate the tokenizer definition to the supported equivalent."],"exampleFix":"# before\ntokenizers:\n  my_tok:\n    text_analyzer:\n      token_filters:\n        - type: stopwordz\n# after\ntokenizers:\n  my_tok:\n    text_analyzer:\n      token_filters:\n        - type: stop_words\n          language: English","handlingStrategy":"validation","validationCode":"// Validate analyzer config before index creation by building the index config\n// against a test index or using quickwit's config validation endpoint.\nfor tok in index_config.tokenizers.values() {\n    tok.text_analyzer().expect(\"invalid tokenizer config\");\n}","typeGuard":null,"tryCatchPattern":"match result {\n    Err(e) if e.to_string().contains(\"failed to build tokenizer\") => {\n        // inspect wrapped cause, fix token_filters entry\n    }\n    other => other?,\n}","preventionTips":["Only use token filter types documented for your Quickwit version.","Lint index configs in CI before applying them.","Re-validate tokenizers after Quickwit upgrades."],"tags":["tokenizer","analyzer","config","validation"],"backgroundTag":"invalid-config-value","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}