{"record":{"id":"8c71e8f411283e2a","repo":"rust-lang/rust-analyzer","slug":"unexpected-token","errorCode":null,"errorMessage":"unexpected token","messagePattern":"unexpected token","errorType":"error_code","errorClass":"ungrammar::parser::Error","httpStatus":null,"severity":"error","filePath":"lib/ungrammar/src/parser.rs","lineNumber":143,"sourceCode":"}\n\nfn seq_rule(p: &mut Parser) -> Result<Rule> {\n    let lhs = atom_rule(p)?;\n\n    let mut seq = vec![lhs];\n    while let Some(rule) = opt_atom_rule(p)? {\n        seq.push(rule)\n    }\n    let res = if seq.len() == 1 { seq.pop().unwrap() } else { Rule::Seq(seq) };\n    Ok(res)\n}\n\nfn atom_rule(p: &mut Parser) -> Result<Rule> {\n    match opt_atom_rule(p)? {\n        Some(it) => Ok(it),\n        None => {\n            let token = p.bump()?;\n            bail!(token.loc, \"unexpected token\")\n        }\n    }\n}\n\nfn opt_atom_rule(p: &mut Parser) -> Result<Option<Rule>> {\n    let token = match p.peek() {\n        Some(it) => it,\n        None => return Ok(None),\n    };\n    let mut res = match &token.kind {\n        TokenKind::Node(name) => {\n            if let Some(lookahead) = p.peek_n(1) {\n                match lookahead.kind {\n                    TokenKind::Eq => return Ok(None),\n                    TokenKind::Colon => {\n                        let label = name.clone();\n                        p.bump()?;\n                        p.bump()?;","sourceCodeStart":125,"sourceCodeEnd":161,"githubUrl":"https://github.com/rust-lang/rust-analyzer/blob/e8f7e90aa3e7b26aa9a000200f606c1078da99ec/lib/ungrammar/src/parser.rs#L125-L161","documentation":"`atom_rule` throws `unexpected token` when the next token cannot start any atom (identifier, string literal, `(`, or `!` opt marker). The parser has consumed the offending token via `p.bump()` and reports its location. This is the generic 'this token does not belong in a rule body' failure of the ungrammar parser.","triggerScenarios":"Parsing a rule body containing a token the grammar language has no production for: stray `=`, `;`, `,`, unclosed `)`, or a numeric/unknown token inside a rule. Raised via `seq_rule`/`opt_atom_rule` whenever `opt_atom_rule` returns None.","commonSituations":"Typos in .ungram files (e.g. writing `Foo, = Bar`), using host-language syntax not supported by ungrammar, editing a rule and leaving dangling punctuation.","solutions":["Check the token at the reported location in the .ungram file and remove or correct it.","Ensure atoms are one of: identifier (node/token reference), quoted string (token), `( ... )` group, or `token?`-style opt.","Run the parser/codegen on the grammar file to iterate until it parses cleanly."],"exampleFix":"// before (.ungram)\nRecordField = 'Name' ':' Expr;\n\n// after (.ungram)\nRecordField = 'Name' ':' Expr","handlingStrategy":"validation","validationCode":"// crude pre-scan: rule bodies should only contain idents, quoted strings, ( ) | ! ?\nfn rule_body_clean(src: &str) -> bool {\n    src.lines().skip_while(|l| !l.contains('=')).all(|l| {\n        l.chars().all(|c| c.is_alphanumeric() || \"_' ()|!?\".contains(c))\n    })\n}","typeGuard":null,"tryCatchPattern":"match ungrammar::parse(src) {\n    Ok(g) => g,\n    Err(e) if e.to_string() == \"unexpected token\" => {\n        eprintln!(\"invalid token in rule body: {}\", e);\n        Err(GrammarError::Syntax(e))\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Only use the supported atom forms: ident, 'quoted token', ( group ), token?.","Strip host-language punctuation (semicolons, commas) when porting EBNF to ungrammar.","Iterate with the codegen parser locally after every grammar edit."],"tags":["parser","ungrammar","syntax"],"backgroundTag":"grammar-parse-error","analyzedSha":"e8f7e90aa3e7b26aa9a000200f606c1078da99ec","analyzedAt":"2026-09-03T21:08:06.959Z","contentChangedAt":"2026-09-03T21:08:06.959Z","schemaVersion":2},"datasetVersion":"2026-09-11T07:07:21.782Z"}