What causes "unclosed token literal" in rust-lang/rust-analyzer?

Add the missing closing single quote to the token literal

unclosed token literal

Error message

unclosed token literal

What it means

The ungrammar lexer raises this while scanning a token literal started with a single quote (`'...'`). If the input ends before a closing quote is found, `chars.next()` returns `None` and the lexer bails. It guards against truncated or malformed grammar files.

Source

Thrown at lib/ungrammar/src/lexer.rs:88

    }
}

fn advance(input: &mut &str) -> Result<TokenKind> {
    let mut chars = input.chars();
    let c = chars.next().unwrap();
    let res = match c {
        '=' => TokenKind::Eq,
        '*' => TokenKind::Star,
        '?' => TokenKind::QMark,
        '(' => TokenKind::LParen,
        ')' => TokenKind::RParen,
        '|' => TokenKind::Pipe,
        ':' => TokenKind::Colon,
        '\'' => {
            let mut buf = String::new();
            loop {
                match chars.next() {
                    None => bail!("unclosed token literal"),
                    Some('\\') => match chars.next() {
                        Some(c) if is_escapable(c) => buf.push(c),
                        _ => bail!("invalid escape in token literal"),
                    },
                    Some('\'') => break,
                    Some(c) => buf.push(c),
                }
            }
            TokenKind::Token(buf)
        }
        c if is_ident_char(c) => {
            let mut buf = String::new();
            buf.push(c);
            loop {
                match chars.clone().next() {
                    Some(c) if is_ident_char(c) => {
                        chars.next();
                        buf.push(c);

View on GitHub (pinned to e8f7e90aa3)

Solutions

Add the missing closing single quote to the token literal
Check the line/offset reported by the lexer to find the unterminated literal
If the quote was intentional as text, note that ungrammar only uses `'` for token literals — remove or escape it

Example fix

// before (grammar.ron / .ungrammar)
Expr = 'binary
// after
Expr = 'binary'

Defensive patterns

Strategy: validation

Validate before calling

// pre-check a .ungrammar file for unterminated token literals
fn has_unterminated_literal(src: &str) -> bool {
    src.chars().filter(|&c| c == '\'').count() % 2 != 0
}

Try / catch

match ungrammar::Grammar::parse(src) {
    Err(e) if e.to_string().contains("unclosed token literal") => {
        eprintln!("grammar file has an unterminated '...' literal: {e}")
    }
    r => r?,
}

Prevention

Count single quotes when hand-editing token literals
Run the grammar parse in CI to catch truncation early
Avoid manual edits near token literals; re-read the file after merges

When it happens

Trigger: A `.ungrammar` file contains `'` starting a token literal that is never closed before end of file, e.g. `Foo = 'bar`.

Common situations: Hand-editing grammar files and deleting the closing quote; copy-paste truncation; merge conflicts leaving partial literals.

Related errors

AI-assisted analysis of rust-lang/rust-analyzer@e8f7e90aa3 (2026-09-03). Data as JSON: /api/errors/f754ff3beebb40dc. Report an issue: GitHub.