{"record":{"id":"facc2529eae783c4","repo":"rust-lang/rust-analyzer","slug":"token-from-lexer-must-be-single-char-token-tok","errorCode":null,"errorMessage":"Token from lexer must be single char: token = {token:#?}","messagePattern":"Token from lexer must be single char: token = (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/syntax-bridge/src/lib.rs","lineNumber":296,"sourceCode":"                    let delim = match kind {\n                        T!['('] => Some(tt::DelimiterKind::Parenthesis),\n                        T!['{'] => Some(tt::DelimiterKind::Brace),\n                        T!['['] => Some(tt::DelimiterKind::Bracket),\n                        _ => None,\n                    };\n\n                    // Start a new subtree\n                    if let Some(kind) = delim {\n                        builder.open(kind, conv.span_for(abs_range));\n                        continue;\n                    }\n\n                    let spacing = match conv.peek().map(|next| next.kind(conv)) {\n                        Some(kind) if is_single_token_op(kind) => tt::Spacing::Joint,\n                        _ => tt::Spacing::Alone,\n                    };\n                    let Some(char) = token.to_char(conv) else {\n                        panic!(\"Token from lexer must be single char: token = {token:#?}\")\n                    };\n                    // FIXME: this might still be an unmatched closing delimiter? Maybe we should assert here\n                    tt::Leaf::from(tt::Punct { char, spacing, span: conv.span_for(abs_range) })\n                }\n                kind => {\n                    macro_rules! make_ident {\n                        () => {\n                            tt::Ident {\n                                span: conv.span_for(abs_range),\n                                sym: Symbol::intern(&token.to_text(conv)),\n                                is_raw: tt::IdentIsRaw::No,\n                            }\n                            .into()\n                        };\n                    }\n                    let leaf: tt::Leaf = match kind {\n                        k if k.is_any_identifier() => {\n                            let text = token.to_text(conv);","sourceCodeStart":278,"sourceCodeEnd":314,"githubUrl":"https://github.com/rust-lang/rust-analyzer/blob/e8f7e90aa3e7b26aa9a000200f606c1078da99ec/crates/syntax-bridge/src/lib.rs#L278-L314","documentation":"When converting punctuation tokens, the converter assumes every lexer token in this arm is a single-character punct and converts it via `token.to_char(conv)`. If `to_char` returns None, the token is a multi-character or unexpected kind masquerading as punctuation, breaking the proc-macro token model (each Punct must be one char), so it panics with a debug dump of the token.","triggerScenarios":"Calling `syntax_node_to_token_tree`/`parse_to_token_tree` (or the _modified/_static_span variants) on input where `is_single_token_op`-style classification admits a token that `to_char` cannot reduce to one char — e.g. after lexer/grammar changes adding multi-char punctuation or a new compound operator reaching this arm.","commonSituations":"Contributors adding new punctuation to the grammar without updating syntax-bridge conversion; parser-level tokens like `..=` or combined operators being fed to the punct path; fuzzed inputs exposing classification gaps.","solutions":["Extend `to_char`/the punct handling to split multi-char tokens into individual single-char Puncts with proper Spacing","Fix the classification predicate so only genuinely single-char tokens take this path","Add a fixture with the offending token to the conversion tests"],"exampleFix":"// before\nlet Some(char) = token.to_char(conv) else {\n    panic!(\"Token from lexer must be single char: token = {token:#?}\")\n};\n// after\nlet chars: Vec<char> = token.to_chars(conv); // split multi-char ops\n// emit one Punct per char, joint spacing between them","handlingStrategy":"validation","validationCode":"// Verify punct tokens are single-char before conversion:\nlet ok = node\n    .descendants_with_tokens()\n    .filter_map(|t| t.into_token())\n    .filter(|t| t.kind() == SyntaxKind::PUNCT)\n    .all(|t| t.text().chars().count() == 1);","typeGuard":null,"tryCatchPattern":"let tt = std::panic::catch_unwind(AssertUnwindSafe(||\n    parse_to_token_tree(&text)\n)).ok()?; // fall back to a conservative re-lex path","preventionTips":["Update syntax-bridge whenever the grammar adds new punctuation","Split multi-char operators into per-char Puncts with Joint spacing early in the pipeline","Add conversion tests for every new multi-char operator"],"tags":["rust","panic","macros","lexer","token-tree"],"backgroundTag":"non-single-char-punct","analyzedSha":"e8f7e90aa3e7b26aa9a000200f606c1078da99ec","analyzedAt":"2026-09-03T21:08:06.959Z","contentChangedAt":"2026-09-03T21:08:06.959Z","schemaVersion":2},"datasetVersion":"2026-09-11T07:07:21.782Z"}