{"record":{"id":"691a90205322612d","repo":"stanfordnlp/CoreNLP","slug":"parsing-failed-error","errorCode":null,"errorMessage":"Parsing failed. Error: ","messagePattern":"Parsing failed\\. Error: ","errorType":"exception","errorClass":"TokenSequenceParseException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/ling/tokensregex/parser/TokenSequenceParser.java","lineNumber":26,"sourceCode":"import edu.stanford.nlp.ling.tokensregex.*;\nimport edu.stanford.nlp.util.CoreMap;\nimport edu.stanford.nlp.util.ArrayMap;\nimport edu.stanford.nlp.util.Pair;\nimport java.io.Reader;\nimport java.io.StringReader;\nimport java.util.*;\nimport java.lang.RuntimeException;\n\npublic class TokenSequenceParser implements SequencePattern.Parser<CoreMap>, TokenSequenceParserConstants {\n    public TokenSequenceParser() {}\n\n    public CoreMapExpressionExtractor getExpressionExtractor(Env env, Reader r) throws ParseException, TokenSequenceParseException {\n        try{\n            TokenSequenceParser p = new TokenSequenceParser(r);\n            List<SequenceMatchRules.Rule> rules = p.RuleList(env);\n            return new CoreMapExpressionExtractor(env, rules);\n        }catch(TokenMgrError error){\n            throw new TokenSequenceParseException(\"Parsing failed. Error: \" + error);\n        }\n    }\n\n    public void updateExpressionExtractor(CoreMapExpressionExtractor extractor, Reader r) throws ParseException, TokenSequenceParseException {\n        try{\n            TokenSequenceParser p = new TokenSequenceParser(r);\n            List<SequenceMatchRules.Rule> rules = p.RuleList(extractor.getEnv());\n            extractor.appendRules(rules);\n        }catch(TokenMgrError error){\n            throw new TokenSequenceParseException(\"Parsing failed. Error: \" + error);\n        }\n    }\n\n        public SequencePattern.PatternExpr parseSequence(Env env, String s) throws ParseException, TokenSequenceParseException {\n        try{\n            TokenSequenceParser p = new TokenSequenceParser(new StringReader(s));\n            return p.SeqRegex(env);\n        }catch(TokenMgrError error){","sourceCodeStart":8,"sourceCodeEnd":44,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/ling/tokensregex/parser/TokenSequenceParser.java#L8-L44","documentation":"TokenSequenceParser.getExpressionExtractor catches TokenMgrError (the JavaCC lexer's error type, an Error not an Exception) while parsing a rules file and rethrows it as TokenSequenceParseException('Parsing failed. Error: ' + error). It means the rule file contains characters/tokens the TokensRegex lexer cannot recognize.","triggerScenarios":"getExpressionExtractor(env, reader) with a Reader over a rule file containing illegal characters, unterminated strings/regex literals, or non-UTF8 bytes that break tokenization.","commonSituations":"TokenRules files edited with smart quotes or invisible Unicode; wrong encoding (e.g., Latin-1 file read as UTF-8); copied rules with curly braces mismatched at the lexer level.","solutions":["Read the wrapped TokenMgrError message for the offending character and line.","Re-save the rules file as plain UTF-8/ASCII, replacing smart quotes and em-dashes with ASCII equivalents.","Check for unterminated string or regex literals in the rules file.","Validate the file against working example rule files shipped with CoreNLP."],"exampleFix":"// before (rules file)\nword: “President”   // smart quotes break the lexer\n// after\nword: \"President\"     // ASCII quotes","handlingStrategy":"try-catch","validationCode":"// pre-scan rules file for characters the lexer rejects\nString text = new String(Files.readAllBytes(path), StandardCharsets.UTF_8);\nfor (int i = 0; i < text.length(); i++) {\n    char c = text.charAt(i);\n    if (c == '\\u201C' || c == '\\u201D' || c == '\\u2018' || c == '\\u2019') {\n        throw new IllegalStateException(\"smart quote at offset \" + i + \" in \" + path);\n    }\n}","typeGuard":null,"tryCatchPattern":"try {\n    CoreMapExpressionExtractor ext = parser.getExpressionExtractor(env, reader);\n} catch (TokenSequenceParseException e) {\n    log.error(\"Rules file lexing failed: \" + e.getMessage());\n}","preventionTips":["Author rules files in UTF-8 with ASCII-only punctuation.","Lint rule files in CI with a trial getExpressionExtractor call.","Open files with explicit StandardCharsets.UTF_8 readers.","Diff new rule files against known-good examples when parse fails."],"tags":["java","tokensregex","lexer","parse-error","rules-file"],"backgroundTag":"invalid-regex-pattern","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}