rust-lang/rust-analyzer · error · ungrammar::parser::Error

unexpected token

Error message

unexpected token

What it means

`atom_rule` throws `unexpected token` when the next token cannot start any atom (identifier, string literal, `(`, or `!` opt marker). The parser has consumed the offending token via `p.bump()` and reports its location. This is the generic 'this token does not belong in a rule body' failure of the ungrammar parser.

Solutions

1. Check the token at the reported location in the .ungram file and remove or correct it.
2. Ensure atoms are one of: identifier (node/token reference), quoted string (token), `( ... )` group, or `token?`-style opt.
3. Run the parser/codegen on the grammar file to iterate until it parses cleanly.

Example fix

// before (.ungram)
RecordField = 'Name' ':' Expr;

// after (.ungram)
RecordField = 'Name' ':' Expr

// crude pre-scan: rule bodies should only contain idents, quoted strings, ( ) | ! ?
fn rule_body_clean(src: &str) -> bool {
    src.lines().skip_while(|l| !l.contains('=')).all(|l| {
        l.chars().all(|c| c.is_alphanumeric() || "_' ()|!?".contains(c))
    })
}

match ungrammar::parse(src) {
    Ok(g) => g,
    Err(e) if e.to_string() == "unexpected token" => {
        eprintln!("invalid token in rule body: {}", e);
        Err(GrammarError::Syntax(e))
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Parsing a rule body containing a token the grammar language has no production for: stray `=`, `;`, `,`, unclosed `)`, or a numeric/unknown token inside a rule. Raised via `seq_rule`/`opt_atom_rule` whenever `opt_atom_rule` returns None.

Common situations: Typos in .ungram files (e.g. writing `Foo, = Bar`), using host-language syntax not supported by ungrammar, editing a rule and leaving dangling punctuation.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• The first element in a sequence of productions or…
• unexpected token, expected
• duplicate rule
• expected at least one token
• expected ident

AI-assisted analysis of rust-lang/rust-analyzer@e8f7e90aa3 (2026-09-03). Data as JSON: /api/errors/8c71e8f411283e2a. Report an issue: GitHub.

Appendix: source

Thrown at lib/ungrammar/src/parser.rs:143

}

fn seq_rule(p: &mut Parser) -> Result<Rule> {
    let lhs = atom_rule(p)?;

    let mut seq = vec![lhs];
    while let Some(rule) = opt_atom_rule(p)? {
        seq.push(rule)
    }
    let res = if seq.len() == 1 { seq.pop().unwrap() } else { Rule::Seq(seq) };
    Ok(res)
}

fn atom_rule(p: &mut Parser) -> Result<Rule> {
    match opt_atom_rule(p)? {
        Some(it) => Ok(it),
        None => {
            let token = p.bump()?;
            bail!(token.loc, "unexpected token")
        }
    }
}

fn opt_atom_rule(p: &mut Parser) -> Result<Option<Rule>> {
    let token = match p.peek() {
        Some(it) => it,
        None => return Ok(None),
    };
    let mut res = match &token.kind {
        TokenKind::Node(name) => {
            if let Some(lookahead) = p.peek_n(1) {
                match lookahead.kind {
                    TokenKind::Eq => return Ok(None),
                    TokenKind::Colon => {
                        let label = name.clone();
                        p.bump()?;
                        p.bump()?;

View on GitHub (pinned to e8f7e90aa3)