{"record":{"id":"d224de3e3488cd23","repo":"quickwit-oss/quickwit","slug":"duplicated-custom-tokenizer","errorCode":null,"errorMessage":"duplicated custom tokenizer: `{}`","messagePattern":"duplicated custom tokenizer: `(.+?)`","errorType":"validation","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs","lineNumber":209,"sourceCode":"            concatenate_dynamic_fields,\n        } = build_mapping_tree(&doc_mapping.field_mappings, &mut schema_builder)?;\n        if !concatenate_dynamic_fields.is_empty() && dynamic_field.is_none() {\n            bail!(\"concatenate field has `include_dynamic_fields` set, but index isn't dynamic\");\n        }\n        let timestamp_field_path = if let Some(timestamp_field_name) = &doc_mapping.timestamp_field\n        {\n            validate_timestamp_field(timestamp_field_name, &field_mappings)?;\n            Some(build_field_path_from_str(timestamp_field_name))\n        } else {\n            None\n        };\n        let schema = schema_builder.build();\n\n        let tokenizer_manager = create_default_quickwit_tokenizer_manager();\n        let mut custom_tokenizer_names = HashSet::new();\n        for tokenizer_config_entry in &doc_mapping.tokenizers {\n            if custom_tokenizer_names.contains(&tokenizer_config_entry.name) {\n                bail!(\n                    \"duplicated custom tokenizer: `{}`\",\n                    tokenizer_config_entry.name\n                );\n            }\n            if tokenizer_manager\n                .get_tokenizer(&tokenizer_config_entry.name)\n                .is_some()\n            {\n                bail!(\n                    \"custom tokenizer name `{}` should be different from built-in tokenizer's \\\n                     names\",\n                    tokenizer_config_entry.name\n                );\n            }\n            let tokenizer = tokenizer_config_entry\n                .config\n                .text_analyzer()\n                .map_err(|error| {","sourceCodeStart":191,"sourceCodeEnd":227,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs#L191-L227","documentation":"While building the tokenizer manager in try_from, each custom tokenizer defined in `doc_mapping.tokenizers` is checked against a HashSet of already-seen names. Defining two custom tokenizers with the same name is ambiguous (which config wins?) so construction fails immediately.","triggerScenarios":"`QuickwitDocMapper::try_from(index_config)` with two entries in `doc_mapping.tokenizers` sharing the same `name` value.","commonSituations":"Merging two index config YAML files or appending tokenizer definitions to an existing config without noticing the name already exists.","solutions":["Rename one of the duplicate custom tokenizers to a unique name.","Delete the redundant tokenizer entry.","Update the field(s) referencing the removed tokenizer to use the surviving one."],"exampleFix":"# before\ntokenizers:\n  - name: my_tok\n    type: ngram\n  - name: my_tok\n    type: raw\n\n# after\ntokenizers:\n  - name: my_ngram_tok\n    type: ngram\n  - name: my_raw_tok\n    type: raw","handlingStrategy":"validation","validationCode":"fn check_dup_tokenizers(cfg: &serde_yaml::Value) -> Result<(), String> {\n    let mut seen = std::collections::HashSet::new();\n    if let Some(toks) = cfg[\"doc_mapping\"][\"tokenizers\"].as_sequence() {\n        for t in toks {\n            let name = t[\"name\"].as_str().unwrap_or_default();\n            if !seen.insert(name) {\n                return Err(format!(\"duplicated custom tokenizer: {name}\"));\n            }\n        }\n    }\n    Ok(())\n}","typeGuard":null,"tryCatchPattern":"match QuickwitDocMapper::try_from(&index_config) {\n    Ok(m) => m,\n    Err(e) if e.to_string().starts_with(\"duplicated custom tokenizer\") => {\n        eprintln!(\"Fix doc_mapping.tokenizers: {e}\");\n        return Err(e);\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Keep a single canonical tokenizer list and reference it, don't merge lists by hand.","Use descriptive, unique tokenizer names (project-prefixed).","Lint configs for duplicate keys in tokenizers lists in CI."],"tags":["config","tokenizer","duplicate"],"backgroundTag":"schema-validation-failed","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}