rust-lang/rust-analyzer · error · ungrammar::parser::Error

unexpected token

Error message

unexpected token

What it means

`atom_rule` throws `unexpected token` when the next token cannot start any atom (identifier, string literal, `(`, or `!` opt marker). The parser has consumed the offending token via `p.bump()` and reports its location. This is the generic 'this token does not belong in a rule body' failure of the ungrammar parser.

Source

Thrown at lib/ungrammar/src/parser.rs:143

}

fn seq_rule(p: &mut Parser) -> Result<Rule> {
    let lhs = atom_rule(p)?;

    let mut seq = vec![lhs];
    while let Some(rule) = opt_atom_rule(p)? {
        seq.push(rule)
    }
    let res = if seq.len() == 1 { seq.pop().unwrap() } else { Rule::Seq(seq) };
    Ok(res)
}

fn atom_rule(p: &mut Parser) -> Result<Rule> {
    match opt_atom_rule(p)? {
        Some(it) => Ok(it),
        None => {
            let token = p.bump()?;
            bail!(token.loc, "unexpected token")
        }
    }
}

fn opt_atom_rule(p: &mut Parser) -> Result<Option<Rule>> {
    let token = match p.peek() {
        Some(it) => it,
        None => return Ok(None),
    };
    let mut res = match &token.kind {
        TokenKind::Node(name) => {
            if let Some(lookahead) = p.peek_n(1) {
                match lookahead.kind {
                    TokenKind::Eq => return Ok(None),
                    TokenKind::Colon => {
                        let label = name.clone();
                        p.bump()?;
                        p.bump()?;

View on GitHub (pinned to e8f7e90aa3)

Solutions

1. Check the token at the reported location in the .ungram file and remove or correct it.
2. Ensure atoms are one of: identifier (node/token reference), quoted string (token), `( ... )` group, or `token?`-style opt.
3. Run the parser/codegen on the grammar file to iterate until it parses cleanly.

Example fix

// before (.ungram)
RecordField = 'Name' ':' Expr;

// after (.ungram)
RecordField = 'Name' ':' Expr

// crude pre-scan: rule bodies should only contain idents, quoted strings, ( ) | ! ?
fn rule_body_clean(src: &str) -> bool {
    src.lines().skip_while(|l| !l.contains('=')).all(|l| {
        l.chars().all(|c| c.is_alphanumeric() || "_' ()|!?".contains(c))
    })
}

match ungrammar::parse(src) {
    Ok(g) => g,
    Err(e) if e.to_string() == "unexpected token" => {
        eprintln!("invalid token in rule body: {}", e);
        Err(GrammarError::Syntax(e))
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Parsing a rule body containing a token the grammar language has no production for: stray `=`, `;`, `,`, unclosed `)`, or a numeric/unknown token inside a rule. Raised via `seq_rule`/`opt_atom_rule` whenever `opt_atom_rule` returns None.

Common situations: Typos in .ungram files (e.g. writing `Foo, = Bar`), using host-language syntax not supported by ungrammar, editing a rule and leaving dangling punctuation.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• unexpected token, expected `{}`
• The first element in a sequence of productions or alternativ
• unclosed token literal
• unexpected character: `{}`
• Undefined node: {}

AI-assisted analysis of rust-lang/rust-analyzer@e8f7e90aa3 (2026-09-03). Data as JSON: /api/errors/8c71e8f411283e2a. Report an issue: GitHub.