What causes "normalizer generated multiple tokens" in quickwit-oss/quickwit?

Remove separator characters (spaces, hyphens) from the wildcard pattern's literal parts.

normalizer generated multiple tokens

Error message

normalizer generated multiple tokens

What it means

Wildcard queries are implemented by normalizing the literal portions of the pattern and matching them as exact tokens within a regex. sub_query_parts_to_regex expects the normalizer to produce exactly one token per literal part; if the token stream yields a second token the pattern cannot be matched deterministically, so it bails with this error.

Solutions

Remove separator characters (spaces, hyphens) from the wildcard pattern's literal parts.
Run wildcard queries against a keyword-type field with a lowercase normalizer instead of a tokenized text field.
Escape or restructure the pattern, or use a regex query where multi-token matching is intended.

Example fix

// before
WildcardQuery { field: "title", pattern: "quick brown*" }
// after
WildcardQuery { field: "title", pattern: "quickbrown*" } // or match against a keyword field

Defensive patterns

Strategy: validation

Validate before calling

if wildcard_pattern.split(|c: char| c.is_whitespace() || c == '-').filter(|s| !s.is_empty()).count() > 1 {
    return Err("wildcard pattern literal contains separators; use a keyword field or simplify the pattern");
}

Try / catch

match build_wildcard_query(pattern) {
    Err(e) if e.to_string().contains("normalizer generated multiple tokens") => {
        // sanitize pattern or switch to a regex/keyword field
    }
    r => r?,
}

Prevention

Avoid spaces and token-separator characters inside wildcard patterns.
Use wildcard queries on keyword fields with simple normalizers (e.g. lowercase).
Test wildcard patterns against the target field's normalizer before deploying.

When it happens

Trigger: Calling to_regex (via sub_query_parts_to_regex) for a WildcardQuery whose literal part, after applying the field's normalizer, produces multiple tokens - typically because the literal contains characters the normalizer splits on, like spaces, hyphens, or other separators, e.g. pattern `foo bar*`.

Common situations: Wildcard patterns containing whitespace or punctuation while the field's normalizer tokenizes them into several tokens; using wildcard queries on text fields tokenized with aggressive analyzers instead of keyword/normalized fields.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/3e07169f504ba946. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-query/src/query_ast/wildcard_query.rs:100

    tokenizer_name: &str,
    tokenizer_manager: &TokenizerManager,
) -> anyhow::Result<String> {
    let mut normalizer = tokenizer_manager
        .get_normalizer(tokenizer_name)
        .with_context(|| format!("no tokenizer named `{tokenizer_name}` is registered"))?;

    sub_query_parts
        .into_iter()
        .map(|part| match part {
            SubQuery::Text(text) => {
                let mut token_stream = normalizer.token_stream(&text);
                let expected_token = token_stream
                    .next()
                    .context("normalizer generated no content")?
                    .text
                    .clone();
                if let Some(_unexpected_token) = token_stream.next() {
                    bail!("normalizer generated multiple tokens")
                }
                Ok(Cow::Owned(regex::escape(&expected_token)))
            }
            SubQuery::Wildcard => Ok(Cow::Borrowed(".*")),
            SubQuery::QuestionMark => Ok(Cow::Borrowed(".")),
        })
        .collect::<Result<String, _>>()
}

impl WildcardQuery {
    pub fn to_regex(
        &self,
        schema: &TantivySchema,
        tokenizer_manager: &TokenizerManager,
    ) -> Result<(Field, Option<Vec<u8>>, String), InvalidQuery> {
        let Some((field, field_entry, json_path)) = find_field_or_hit_dynamic(&self.field, schema)
        else {
            return Err(InvalidQuery::FieldDoesNotExist {

View on GitHub (pinned to a39730c5cd)