rust-lang/rust-analyzer · error

invalid escape in token literal

Error message

invalid escape in token literal

What it means

Inside a `'...'` token literal, a backslash must be followed by an escapable character (checked by `is_escapable`). Any other escape sequence, or a trailing lone backslash at end of input, bails with this error.

Source

Thrown at lib/ungrammar/src/lexer.rs:91

fn advance(input: &mut &str) -> Result<TokenKind> {
    let mut chars = input.chars();
    let c = chars.next().unwrap();
    let res = match c {
        '=' => TokenKind::Eq,
        '*' => TokenKind::Star,
        '?' => TokenKind::QMark,
        '(' => TokenKind::LParen,
        ')' => TokenKind::RParen,
        '|' => TokenKind::Pipe,
        ':' => TokenKind::Colon,
        '\'' => {
            let mut buf = String::new();
            loop {
                match chars.next() {
                    None => bail!("unclosed token literal"),
                    Some('\\') => match chars.next() {
                        Some(c) if is_escapable(c) => buf.push(c),
                        _ => bail!("invalid escape in token literal"),
                    },
                    Some('\'') => break,
                    Some(c) => buf.push(c),
                }
            }
            TokenKind::Token(buf)
        }
        c if is_ident_char(c) => {
            let mut buf = String::new();
            buf.push(c);
            loop {
                match chars.clone().next() {
                    Some(c) if is_ident_char(c) => {
                        chars.next();
                        buf.push(c);
                    }
                    _ => break,
                }

View on GitHub (pinned to e8f7e90aa3)

Solutions

1. Remove the backslash and use the literal character directly if it needs no escaping
2. Only escape characters accepted by `is_escapable` (e.g. `\'` and `\\`)
3. Replace control characters with the grammar's own notation or restructure the token

Example fix

// before
Token = 'line\nend'
// after
Token = 'line\nend'  // or drop the escape: 'line end'

// reject escapes not accepted by ungrammar's is_escapable
fn has_bad_escape(src: &str) -> bool {
    let esc: Vec<char> = vec!['\'', '\\'];
    let mut it = src.chars().peekable();
    while let Some(c) = it.next() {
        if c == '\\' {
            match it.peek() {
                None => return true,
                Some(&n) if !esc.contains(&n) => return true,
                _ => {}
            }
        }
    }
    false
}

match ungrammar::Grammar::parse(src) {
    Err(e) if e.to_string().contains("invalid escape") => {
        eprintln!("unsupported escape in token literal: {e}")
    }
    r => r?,
}

When it happens

Trigger: Writing an unsupported escape such as `'\n'`, `'\t'`, or `\` immediately before the end of input inside a token literal in a `.ungrammar` file.

Common situations: Authors assuming C-style escapes (\n, \t, \u...) work in ungrammar; regex-style escapes copied from other DSLs.

Related errors

• unclosed token literal
• unexpected character: `{}`
• unexpected `\r`, only Unix-style line endings allowed
• unexpected token, expected `{}`
• Token from lexer must be single char: token = {token:#?}

AI-assisted analysis of rust-lang/rust-analyzer@e8f7e90aa3 (2026-09-03). Data as JSON: /api/errors/b5e649ae5d586448. Report an issue: GitHub.