updating tokenizers is allowed only if adding new tokenizers

Error message

updating tokenizers is allowed only if adding new tokenizers, current tokenizers `{current_tokenizers:?}`, new tokenizers `{new_tokenizers:?}`

What it means

Tokenizer definitions referenced by field mappings may change between updates, but removing or altering existing tokenizers would invalidate how previously indexed splits were analyzed. prepare_doc_mapping_update only allows the new tokenizer set to be a superset of the current one; any removal/shrink fails with this error.

Source

Thrown at quickwit/quickwit-config/src/index_config/mod.rs:746

    ensure!(
        new_doc_mapping.doc_mapping_uid != current_doc_mapping.doc_mapping_uid,
        "new doc mapping UID should differ from the current one, current UID `{}`, new UID `{}`",
        current_doc_mapping.doc_mapping_uid,
        new_doc_mapping.doc_mapping_uid,
    );
    let new_timestamp_field = new_doc_mapping.timestamp_field.as_deref();
    let current_timestamp_field = current_doc_mapping.timestamp_field.as_deref();
    ensure!(
        new_timestamp_field == current_timestamp_field,
        "updating timestamp field is not allowed, current timestamp field `{}`, new timestamp \
         field `{}`",
        current_timestamp_field.unwrap_or("none"),
        new_timestamp_field.unwrap_or("none"),
    );
    // TODO: Unsure this constraint is required, should we relax it?
    let new_tokenizers: HashSet<_> = new_doc_mapping.tokenizers.iter().collect();
    let current_tokenizers: HashSet<_> = current_doc_mapping.tokenizers.iter().collect();
    ensure!(
        new_tokenizers.is_superset(&current_tokenizers),
        "updating tokenizers is allowed only if adding new tokenizers, current tokenizers \
         `{current_tokenizers:?}`, new tokenizers `{new_tokenizers:?}`",
    );
    build_doc_mapper(&new_doc_mapping, search_settings).context("invalid doc mapping")?;
    Ok((new_doc_mapping, true))
}

#[cfg(test)]
mod tests {

    use cron::TimeUnitSpec;
    use quickwit_doc_mapper::{Mode, ModeType, TokenizerEntry};
    use quickwit_proto::types::DocMappingUid;

    use super::*;
    use crate::ConfigFormat;
    use crate::merge_policy_config::MergePolicyConfig;

View on GitHub (pinned to a39730c5cd)

Solutions

Keep all existing tokenizer definitions and only add new ones in the update
If a tokenizer must be removed/changed, create a new index and reindex the data
Restore the missing tokenizer entries so the new set is a superset of the current one

Example fix

# before (update)
tokenizers: [default]            # removed custom_tokenizer
# after
tokenizers: [default, custom_tokenizer, new_tokenizer]  # superset only

Defensive patterns

Strategy: validation

Validate before calling

// Rust precheck
let cur: HashSet<&str> = current.tokenizers.iter().map(|s| s.as_str()).collect();
let new: HashSet<&str> = new_m.tokenizers.iter().map(|s| s.as_str()).collect();
assert!(new.is_superset(&cur), "tokenizers may only be added, not removed");

Prevention

Only append new tokenizers in mapping updates
Don't rename tokenizers; add the new name and keep the old one
Track tokenizer changes in code review when editing index configs

When it happens

Trigger: Submitting a doc mapping update whose `tokenizers` set drops an existing tokenizer name, or replaces a tokenizer entry such that the new set is not a superset of the current set.

Common situations: Cleaning up 'unused' tokenizers in an index config update; renaming a custom tokenizer (old name disappears from the set); reverting a config to an older version that had fewer tokenizers.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/ea44423344970012. Report an issue: GitHub.