{"record":{"id":"05a6da50ab0e8130","repo":"stanfordnlp/CoreNLP","slug":"parsing-failed-error-05a6da","errorCode":null,"errorMessage":"Parsing failed. Error: ","messagePattern":"Parsing failed\\. Error: ","errorType":"exception","errorClass":"TokenSequenceParseException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/ling/tokensregex/parser/TokenSequenceParser.jj","lineNumber":35,"sourceCode":"import edu.stanford.nlp.ling.tokensregex.*;\nimport edu.stanford.nlp.util.CoreMap;\nimport edu.stanford.nlp.util.ArrayMap;\nimport edu.stanford.nlp.util.Pair;\nimport java.io.Reader;\nimport java.io.StringReader;\nimport java.util.*;\nimport java.lang.RuntimeException;\n\npublic class TokenSequenceParser implements SequencePattern.Parser<CoreMap> {\n    public TokenSequenceParser() {}\n\n    public CoreMapExpressionExtractor getExpressionExtractor(Env env, Reader r) throws ParseException, TokenSequenceParseException {\n        try{\n            TokenSequenceParser p = new TokenSequenceParser(r);\n            List<SequenceMatchRules.Rule> rules = p.RuleList(env);\n            return new CoreMapExpressionExtractor(env, rules);\n        }catch(TokenMgrError error){\n            throw new TokenSequenceParseException(\"Parsing failed. Error: \" + error);\n        }\n    }\n\n    public void updateExpressionExtractor(CoreMapExpressionExtractor extractor, Reader r) throws ParseException, TokenSequenceParseException {\n        try{\n            TokenSequenceParser p = new TokenSequenceParser(r);\n            List<SequenceMatchRules.Rule> rules = p.RuleList(extractor.getEnv());\n            extractor.appendRules(rules);\n        }catch(TokenMgrError error){\n            throw new TokenSequenceParseException(\"Parsing failed. Error: \" + error);\n        }\n    }\n\n\tpublic SequencePattern.PatternExpr parseSequence(Env env, String s) throws ParseException, TokenSequenceParseException {\n        try{\n            TokenSequenceParser p = new TokenSequenceParser(new StringReader(s));\n            return p.SeqRegex(env);\n        }catch(TokenMgrError error){","sourceCodeStart":17,"sourceCodeEnd":53,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/ling/tokensregex/parser/TokenSequenceParser.jj#L17-L53","documentation":"TokenSequenceParser.getExpressionExtractor wraps TokenMgrError (thrown by the generated JavaCC lexer when it encounters characters/sequences that cannot be tokenized under the TokensRegex grammar) into a TokenSequenceParseException with the message \"Parsing failed. Error: <detail>\". It indicates the rules file being read is lexically invalid, so no expression extractor can be built from it.","triggerScenarios":"Calling TokenSequenceParser.getExpressionExtractor(env, reader) with a Reader over a TokensRegex rules file containing an illegal character or malformed token; similarly updateExpressionExtractor. Note that ParseException from p.RuleList(env) is NOT wrapped — only TokenMgrError is.","commonSituations":"Typo or stray character (unmatched quote, invalid escape, control character) in a TokensRegex .rules file loaded for CoreNLP's TokensRegexAnnotator or CoreMapExpressionExtractor; encoding issues producing non-ASCII garbage; rule files edited by hand.","solutions":["Read the wrapped TokenMgrError detail in the message; it pinpoints the offending line/column and character in the rules file.","Fix the illegal character or malformed token at that location (unbalanced quotes, invalid escapes, stray symbols).","Re-save the rules file as UTF-8 without BOM and ensure the Reader uses the matching charset.","Catch TokenSequenceParseException around getExpressionExtractor and fail fast with the file path in your own message for diagnosability."],"exampleFix":"// before\nReader r = new InputStreamReader(new FileInputStream(rulesFile)); // default charset, possible mojibake\nCoreMapExpressionExtractor ex = new TokenSequenceParser().getExpressionExtractor(env, r);\n// after\nReader r = new InputStreamReader(new FileInputStream(rulesFile), StandardCharsets.UTF_8);\nCoreMapExpressionExtractor ex = new TokenSequenceParser().getExpressionExtractor(env, r);","handlingStrategy":"try-catch","validationCode":"// pre-flight: check file exists, is UTF-8, and has no obviously stray control chars\nbyte[] bytes = java.nio.file.Files.readAllBytes(rulesPath);\nString text = new String(bytes, StandardCharsets.UTF_8);\nfor (int i = 0; i < text.length(); i++) {\n  char c = text.charAt(i);\n  if (Character.isISOControl(c) && c != '\\n' && c != '\\r' && c != '\\t')\n    throw new IllegalStateException(\"Illegal control char U+\" + Integer.toHexString(c) + \" at offset \" + i);\n}","typeGuard":null,"tryCatchPattern":"try {\n  extractor = new TokenSequenceParser().getExpressionExtractor(env, reader);\n} catch (TokenSequenceParseException e) {\n  logger.error(\"TokensRegex rules file \" + rulesPath + \" failed to lex/parse: \" + e.getMessage());\n  throw e;\n} catch (ParseException e) {\n  logger.error(\"TokensRegex grammar error in \" + rulesPath + \": \" + e.getMessage());\n  throw e;\n}","preventionTips":["Always read rules files as UTF-8 explicitly.","Validate rules files in CI by constructing the extractor before deployment.","Fix unbalanced quotes and invalid escapes at the line/column reported inside the message.","Remember only TokenMgrError is wrapped; catch ParseException separately for grammar errors."],"tags":["java","parsing","tokensregex","javacc"],"backgroundTag":"json-parse-error","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}