{"record":{"id":"ea44423344970012","repo":"quickwit-oss/quickwit","slug":"updating-tokenizers-is-allowed-only-if-adding-new","errorCode":null,"errorMessage":"updating tokenizers is allowed only if adding new tokenizers, current tokenizers `{current_tokenizers:?}`, new tokenizers `{new_tokenizers:?}`","messagePattern":"updating tokenizers is allowed only if adding new tokenizers, current tokenizers `(.+?)`, new tokenizers `(.+?)`","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-config/src/index_config/mod.rs","lineNumber":746,"sourceCode":"    ensure!(\n        new_doc_mapping.doc_mapping_uid != current_doc_mapping.doc_mapping_uid,\n        \"new doc mapping UID should differ from the current one, current UID `{}`, new UID `{}`\",\n        current_doc_mapping.doc_mapping_uid,\n        new_doc_mapping.doc_mapping_uid,\n    );\n    let new_timestamp_field = new_doc_mapping.timestamp_field.as_deref();\n    let current_timestamp_field = current_doc_mapping.timestamp_field.as_deref();\n    ensure!(\n        new_timestamp_field == current_timestamp_field,\n        \"updating timestamp field is not allowed, current timestamp field `{}`, new timestamp \\\n         field `{}`\",\n        current_timestamp_field.unwrap_or(\"none\"),\n        new_timestamp_field.unwrap_or(\"none\"),\n    );\n    // TODO: Unsure this constraint is required, should we relax it?\n    let new_tokenizers: HashSet<_> = new_doc_mapping.tokenizers.iter().collect();\n    let current_tokenizers: HashSet<_> = current_doc_mapping.tokenizers.iter().collect();\n    ensure!(\n        new_tokenizers.is_superset(&current_tokenizers),\n        \"updating tokenizers is allowed only if adding new tokenizers, current tokenizers \\\n         `{current_tokenizers:?}`, new tokenizers `{new_tokenizers:?}`\",\n    );\n    build_doc_mapper(&new_doc_mapping, search_settings).context(\"invalid doc mapping\")?;\n    Ok((new_doc_mapping, true))\n}\n\n#[cfg(test)]\nmod tests {\n\n    use cron::TimeUnitSpec;\n    use quickwit_doc_mapper::{Mode, ModeType, TokenizerEntry};\n    use quickwit_proto::types::DocMappingUid;\n\n    use super::*;\n    use crate::ConfigFormat;\n    use crate::merge_policy_config::MergePolicyConfig;","sourceCodeStart":728,"sourceCodeEnd":764,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-config/src/index_config/mod.rs#L728-L764","documentation":"Tokenizer definitions referenced by field mappings may change between updates, but removing or altering existing tokenizers would invalidate how previously indexed splits were analyzed. prepare_doc_mapping_update only allows the new tokenizer set to be a superset of the current one; any removal/shrink fails with this error.","triggerScenarios":"Submitting a doc mapping update whose `tokenizers` set drops an existing tokenizer name, or replaces a tokenizer entry such that the new set is not a superset of the current set.","commonSituations":"Cleaning up 'unused' tokenizers in an index config update; renaming a custom tokenizer (old name disappears from the set); reverting a config to an older version that had fewer tokenizers.","solutions":["Keep all existing tokenizer definitions and only add new ones in the update","If a tokenizer must be removed/changed, create a new index and reindex the data","Restore the missing tokenizer entries so the new set is a superset of the current one"],"exampleFix":"# before (update)\ntokenizers: [default]            # removed custom_tokenizer\n# after\ntokenizers: [default, custom_tokenizer, new_tokenizer]  # superset only","handlingStrategy":"validation","validationCode":"// Rust precheck\nlet cur: HashSet<&str> = current.tokenizers.iter().map(|s| s.as_str()).collect();\nlet new: HashSet<&str> = new_m.tokenizers.iter().map(|s| s.as_str()).collect();\nassert!(new.is_superset(&cur), \"tokenizers may only be added, not removed\");","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Only append new tokenizers in mapping updates","Don't rename tokenizers; add the new name and keep the old one","Track tokenizer changes in code review when editing index configs"],"tags":["config","schema","tokenizers","validation"],"backgroundTag":"unsupported-operation","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}