rust-lang/rust-analyzer · error

invalid escape in token literal

Error message

invalid escape in token literal

What it means

Inside a `'...'` token literal, a backslash must be followed by an escapable character (checked by `is_escapable`). Any other escape sequence, or a trailing lone backslash at end of input, bails with this error.

Solutions

1. Remove the backslash and use the literal character directly if it needs no escaping
2. Only escape characters accepted by `is_escapable` (e.g. `\'` and `\\`)
3. Replace control characters with the grammar's own notation or restructure the token

Example fix

// before
Token = 'line\nend'
// after
Token = 'line\nend'  // or drop the escape: 'line end'

// reject escapes not accepted by ungrammar's is_escapable
fn has_bad_escape(src: &str) -> bool {
    let esc: Vec<char> = vec!['\'', '\\'];
    let mut it = src.chars().peekable();
    while let Some(c) = it.next() {
        if c == '\\' {
            match it.peek() {
                None => return true,
                Some(&n) if !esc.contains(&n) => return true,
                _ => {}
            }
        }
    }
    false
}

match ungrammar::Grammar::parse(src) {
    Err(e) if e.to_string().contains("invalid escape") => {
        eprintln!("unsupported escape in token literal: {e}")
    }
    r => r?,
}

When it happens

Trigger: Writing an unsupported escape such as `'\n'`, `'\t'`, or `\` immediately before the end of input inside a token literal in a `.ungrammar` file.

Common situations: Authors assuming C-style escapes (\n, \t, \u...) work in ungrammar; regex-style escapes copied from other DSLs.

Related errors

• unclosed token literal
• unexpected character
• unexpected `\r`, only Unix-style line endings allowed
• unexpected token, expected
• duplicate rule

AI-assisted analysis of rust-lang/rust-analyzer@e8f7e90aa3 (2026-09-03). Data as JSON: /api/errors/b5e649ae5d586448. Report an issue: GitHub.

Appendix: source

Thrown at lib/ungrammar/src/lexer.rs:91

fn advance(input: &mut &str) -> Result<TokenKind> {
    let mut chars = input.chars();
    let c = chars.next().unwrap();
    let res = match c {
        '=' => TokenKind::Eq,
        '*' => TokenKind::Star,
        '?' => TokenKind::QMark,
        '(' => TokenKind::LParen,
        ')' => TokenKind::RParen,
        '|' => TokenKind::Pipe,
        ':' => TokenKind::Colon,
        '\'' => {
            let mut buf = String::new();
            loop {
                match chars.next() {
                    None => bail!("unclosed token literal"),
                    Some('\\') => match chars.next() {
                        Some(c) if is_escapable(c) => buf.push(c),
                        _ => bail!("invalid escape in token literal"),
                    },
                    Some('\'') => break,
                    Some(c) => buf.push(c),
                }
            }
            TokenKind::Token(buf)
        }
        c if is_ident_char(c) => {
            let mut buf = String::new();
            buf.push(c);
            loop {
                match chars.clone().next() {
                    Some(c) if is_ident_char(c) => {
                        chars.next();
                        buf.push(c);
                    }
                    _ => break,
                }

View on GitHub (pinned to e8f7e90aa3)