rust-lang/rust-analyzer · error

Token from lexer must be single char: token = {token:#?}

Error message

Token from lexer must be single char: token = {token:#?}

What it means

When converting punctuation tokens, the converter assumes every lexer token in this arm is a single-character punct and converts it via `token.to_char(conv)`. If `to_char` returns None, the token is a multi-character or unexpected kind masquerading as punctuation, breaking the proc-macro token model (each Punct must be one char), so it panics with a debug dump of the token.

Source

Thrown at crates/syntax-bridge/src/lib.rs:296

                    let delim = match kind {
                        T!['('] => Some(tt::DelimiterKind::Parenthesis),
                        T!['{'] => Some(tt::DelimiterKind::Brace),
                        T!['['] => Some(tt::DelimiterKind::Bracket),
                        _ => None,
                    };

                    // Start a new subtree
                    if let Some(kind) = delim {
                        builder.open(kind, conv.span_for(abs_range));
                        continue;
                    }

                    let spacing = match conv.peek().map(|next| next.kind(conv)) {
                        Some(kind) if is_single_token_op(kind) => tt::Spacing::Joint,
                        _ => tt::Spacing::Alone,
                    };
                    let Some(char) = token.to_char(conv) else {
                        panic!("Token from lexer must be single char: token = {token:#?}")
                    };
                    // FIXME: this might still be an unmatched closing delimiter? Maybe we should assert here
                    tt::Leaf::from(tt::Punct { char, spacing, span: conv.span_for(abs_range) })
                }
                kind => {
                    macro_rules! make_ident {
                        () => {
                            tt::Ident {
                                span: conv.span_for(abs_range),
                                sym: Symbol::intern(&token.to_text(conv)),
                                is_raw: tt::IdentIsRaw::No,
                            }
                            .into()
                        };
                    }
                    let leaf: tt::Leaf = match kind {
                        k if k.is_any_identifier() => {
                            let text = token.to_text(conv);

View on GitHub (pinned to e8f7e90aa3)

Solutions

1. Extend `to_char`/the punct handling to split multi-char tokens into individual single-char Puncts with proper Spacing
2. Fix the classification predicate so only genuinely single-char tokens take this path
3. Add a fixture with the offending token to the conversion tests

Example fix

// before
let Some(char) = token.to_char(conv) else {
    panic!("Token from lexer must be single char: token = {token:#?}")
};
// after
let chars: Vec<char> = token.to_chars(conv); // split multi-char ops
// emit one Punct per char, joint spacing between them

// Verify punct tokens are single-char before conversion:
let ok = node
    .descendants_with_tokens()
    .filter_map(|t| t.into_token())
    .filter(|t| t.kind() == SyntaxKind::PUNCT)
    .all(|t| t.text().chars().count() == 1);

let tt = std::panic::catch_unwind(AssertUnwindSafe(||
    parse_to_token_tree(&text)
)).ok()?; // fall back to a conservative re-lex path

When it happens

Trigger: Calling `syntax_node_to_token_tree`/`parse_to_token_tree` (or the _modified/_static_span variants) on input where `is_single_token_op`-style classification admits a token that `to_char` cannot reduce to one char — e.g. after lexer/grammar changes adding multi-char punctuation or a new compound operator reaching this arm.

Common situations: Contributors adding new punctuation to the grammar without updating syntax-bridge conversion; parser-level tokens like `..=` or combined operators being fed to the punct path; fuzzed inputs exposing classification gaps.

Related errors

• unmatched closing delimiter from syntax fixup
• Next token must be ident
• We explicitly do not provide canonicalization API, as that i
• bad kind {other}
• bad spacing {other}

AI-assisted analysis of rust-lang/rust-analyzer@e8f7e90aa3 (2026-09-03). Data as JSON: /api/errors/facc2529eae783c4. Report an issue: GitHub.