{"record":{"id":"4a89c2408e325c08","repo":"quickwit-oss/quickwit","slug":"custom-tokenizer-name-should-be-different-fro","errorCode":null,"errorMessage":"custom tokenizer name `{}` should be different from built-in tokenizer's names","messagePattern":"custom tokenizer name `(.+?)` should be different from built-in tokenizer's names","errorType":"validation","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs","lineNumber":218,"sourceCode":"        } else {\n            None\n        };\n        let schema = schema_builder.build();\n\n        let tokenizer_manager = create_default_quickwit_tokenizer_manager();\n        let mut custom_tokenizer_names = HashSet::new();\n        for tokenizer_config_entry in &doc_mapping.tokenizers {\n            if custom_tokenizer_names.contains(&tokenizer_config_entry.name) {\n                bail!(\n                    \"duplicated custom tokenizer: `{}`\",\n                    tokenizer_config_entry.name\n                );\n            }\n            if tokenizer_manager\n                .get_tokenizer(&tokenizer_config_entry.name)\n                .is_some()\n            {\n                bail!(\n                    \"custom tokenizer name `{}` should be different from built-in tokenizer's \\\n                     names\",\n                    tokenizer_config_entry.name\n                );\n            }\n            let tokenizer = tokenizer_config_entry\n                .config\n                .text_analyzer()\n                .map_err(|error| {\n                    anyhow::anyhow!(\n                        \"failed to build tokenizer `{}`: {:?}\",\n                        tokenizer_config_entry.name,\n                        error\n                    )\n                })?;\n            let does_lowercasing = tokenizer_config_entry\n                .config\n                .filters","sourceCodeStart":200,"sourceCodeEnd":236,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs#L200-L236","documentation":"Custom tokenizer names are validated against Quickwit's built-in tokenizer set (default, raw, en_stem, etc.) before registration. Shadowing a built-in name would create ambiguity about which tokenizer is used, so it is rejected.","triggerScenarios":"`QuickwitDocMapper::try_from` where an entry in `doc_mapping.tokenizers` has a `name` matching a tokenizer already registered in the default Quickwit tokenizer manager.","commonSituations":"Naming a custom tokenizer `default` or `raw` assuming custom names take a separate namespace; porting configs from systems where such names are allowed.","solutions":["Rename the custom tokenizer to something not colliding with built-in names (e.g. prefix it: `my_default`).","If the built-in behavior is wanted, remove the custom entry and reference the built-in tokenizer directly in field options."],"exampleFix":"# before\ntokenizers:\n  - name: raw\n    type: ngram\n\n# after\ntokenizers:\n  - name: custom_raw_ngram\n    type: ngram","handlingStrategy":"validation","validationCode":"const BUILTIN_TOKENIZERS: [&str; 7] = [\"raw\", \"default\", \"en_stem\", \"whitespace\", \"liju\", \"chinese_compatible\", \"multilang\"];\nfn check_builtin_collision(cfg: &serde_yaml::Value) -> Result<(), String> {\n    if let Some(toks) = cfg[\"doc_mapping\"][\"tokenizers\"].as_sequence() {\n        for t in toks {\n            let name = t[\"name\"].as_str().unwrap_or_default();\n            if BUILTIN_TOKENIZERS.contains(&name) {\n                return Err(format!(\"custom tokenizer name `{name}` collides with a built-in\"));\n            }\n        }\n    }\n    Ok(())\n}","typeGuard":null,"tryCatchPattern":"if let Err(e) = QuickwitDocMapper::try_from(&index_config) {\n    if e.to_string().contains(\"built-in tokenizer\") {\n        eprintln!(\"Rename the custom tokenizer to avoid a built-in name: {e}\");\n    }\n    return Err(e);\n}","preventionTips":["Never name custom tokenizers after built-ins (`raw`, `default`, language tokenizers).","Prefix custom tokenizer names with a project or team prefix.","Check the tokenizer manager docs for the built-in name list before naming."],"tags":["config","tokenizer","naming-conflict"],"backgroundTag":"invalid-config-value","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}