{"record":{"id":"3e07169f504ba946","repo":"quickwit-oss/quickwit","slug":"normalizer-generated-multiple-tokens","errorCode":null,"errorMessage":"normalizer generated multiple tokens","messagePattern":"normalizer generated multiple tokens","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-query/src/query_ast/wildcard_query.rs","lineNumber":100,"sourceCode":"    tokenizer_name: &str,\n    tokenizer_manager: &TokenizerManager,\n) -> anyhow::Result<String> {\n    let mut normalizer = tokenizer_manager\n        .get_normalizer(tokenizer_name)\n        .with_context(|| format!(\"no tokenizer named `{tokenizer_name}` is registered\"))?;\n\n    sub_query_parts\n        .into_iter()\n        .map(|part| match part {\n            SubQuery::Text(text) => {\n                let mut token_stream = normalizer.token_stream(&text);\n                let expected_token = token_stream\n                    .next()\n                    .context(\"normalizer generated no content\")?\n                    .text\n                    .clone();\n                if let Some(_unexpected_token) = token_stream.next() {\n                    bail!(\"normalizer generated multiple tokens\")\n                }\n                Ok(Cow::Owned(regex::escape(&expected_token)))\n            }\n            SubQuery::Wildcard => Ok(Cow::Borrowed(\".*\")),\n            SubQuery::QuestionMark => Ok(Cow::Borrowed(\".\")),\n        })\n        .collect::<Result<String, _>>()\n}\n\nimpl WildcardQuery {\n    pub fn to_regex(\n        &self,\n        schema: &TantivySchema,\n        tokenizer_manager: &TokenizerManager,\n    ) -> Result<(Field, Option<Vec<u8>>, String), InvalidQuery> {\n        let Some((field, field_entry, json_path)) = find_field_or_hit_dynamic(&self.field, schema)\n        else {\n            return Err(InvalidQuery::FieldDoesNotExist {","sourceCodeStart":82,"sourceCodeEnd":118,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-query/src/query_ast/wildcard_query.rs#L82-L118","documentation":"Wildcard queries are implemented by normalizing the literal portions of the pattern and matching them as exact tokens within a regex. sub_query_parts_to_regex expects the normalizer to produce exactly one token per literal part; if the token stream yields a second token the pattern cannot be matched deterministically, so it bails with this error.","triggerScenarios":"Calling to_regex (via sub_query_parts_to_regex) for a WildcardQuery whose literal part, after applying the field's normalizer, produces multiple tokens - typically because the literal contains characters the normalizer splits on, like spaces, hyphens, or other separators, e.g. pattern `foo bar*`.","commonSituations":"Wildcard patterns containing whitespace or punctuation while the field's normalizer tokenizes them into several tokens; using wildcard queries on text fields tokenized with aggressive analyzers instead of keyword/normalized fields.","solutions":["Remove separator characters (spaces, hyphens) from the wildcard pattern's literal parts.","Run wildcard queries against a keyword-type field with a lowercase normalizer instead of a tokenized text field.","Escape or restructure the pattern, or use a regex query where multi-token matching is intended."],"exampleFix":"// before\nWildcardQuery { field: \"title\", pattern: \"quick brown*\" }\n// after\nWildcardQuery { field: \"title\", pattern: \"quickbrown*\" } // or match against a keyword field","handlingStrategy":"validation","validationCode":"if wildcard_pattern.split(|c: char| c.is_whitespace() || c == '-').filter(|s| !s.is_empty()).count() > 1 {\n    return Err(\"wildcard pattern literal contains separators; use a keyword field or simplify the pattern\");\n}","typeGuard":null,"tryCatchPattern":"match build_wildcard_query(pattern) {\n    Err(e) if e.to_string().contains(\"normalizer generated multiple tokens\") => {\n        // sanitize pattern or switch to a regex/keyword field\n    }\n    r => r?,\n}","preventionTips":["Avoid spaces and token-separator characters inside wildcard patterns.","Use wildcard queries on keyword fields with simple normalizers (e.g. lowercase).","Test wildcard patterns against the target field's normalizer before deploying."],"tags":["wildcard-query","normalizer","tokenization"],"backgroundTag":"invalid-argument-value","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}