stanfordnlp/CoreNLP · error · TokenSequenceParseException

Parsing failed. Error:

Error message

Parsing failed. Error: 

What it means

TokenSequenceParser.getExpressionExtractor wraps TokenMgrError (thrown by the generated JavaCC lexer when it encounters characters/sequences that cannot be tokenized under the TokensRegex grammar) into a TokenSequenceParseException with the message "Parsing failed. Error: <detail>". It indicates the rules file being read is lexically invalid, so no expression extractor can be built from it.

Solutions

1. Read the wrapped TokenMgrError detail in the message; it pinpoints the offending line/column and character in the rules file.
2. Fix the illegal character or malformed token at that location (unbalanced quotes, invalid escapes, stray symbols).
3. Re-save the rules file as UTF-8 without BOM and ensure the Reader uses the matching charset.
4. Catch TokenSequenceParseException around getExpressionExtractor and fail fast with the file path in your own message for diagnosability.

Example fix

// before
Reader r = new InputStreamReader(new FileInputStream(rulesFile)); // default charset, possible mojibake
CoreMapExpressionExtractor ex = new TokenSequenceParser().getExpressionExtractor(env, r);
// after
Reader r = new InputStreamReader(new FileInputStream(rulesFile), StandardCharsets.UTF_8);
CoreMapExpressionExtractor ex = new TokenSequenceParser().getExpressionExtractor(env, r);

// pre-flight: check file exists, is UTF-8, and has no obviously stray control chars
byte[] bytes = java.nio.file.Files.readAllBytes(rulesPath);
String text = new String(bytes, StandardCharsets.UTF_8);
for (int i = 0; i < text.length(); i++) {
  char c = text.charAt(i);
  if (Character.isISOControl(c) && c != '\n' && c != '\r' && c != '\t')
    throw new IllegalStateException("Illegal control char U+" + Integer.toHexString(c) + " at offset " + i);
}

try {
  extractor = new TokenSequenceParser().getExpressionExtractor(env, reader);
} catch (TokenSequenceParseException e) {
  logger.error("TokensRegex rules file " + rulesPath + " failed to lex/parse: " + e.getMessage());
  throw e;
} catch (ParseException e) {
  logger.error("TokensRegex grammar error in " + rulesPath + ": " + e.getMessage());
  throw e;
}

When it happens

Trigger: Calling TokenSequenceParser.getExpressionExtractor(env, reader) with a Reader over a TokensRegex rules file containing an illegal character or malformed token; similarly updateExpressionExtractor. Note that ParseException from p.RuleList(env) is NOT wrapped — only TokenMgrError is.

Common situations: Typo or stray character (unmatched quote, invalid escape, control character) in a TokensRegex .rules file loaded for CoreNLP's TokensRegexAnnotator or CoreMapExpressionExtractor; encoding issues producing non-ASCII garbage; rule files edited by hand.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

Related errors

• Error extracting annotation from
• Bad data format:
• Bad number put into wordToNumber. Word is: \"" + input +…
• Error in wordToNumber function.
• Bad number put into wordToNumber. Word is: \"" + curPart +…

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/05a6da50ab0e8130. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/ling/tokensregex/parser/TokenSequenceParser.jj:35

import edu.stanford.nlp.ling.tokensregex.*;
import edu.stanford.nlp.util.CoreMap;
import edu.stanford.nlp.util.ArrayMap;
import edu.stanford.nlp.util.Pair;
import java.io.Reader;
import java.io.StringReader;
import java.util.*;
import java.lang.RuntimeException;

public class TokenSequenceParser implements SequencePattern.Parser<CoreMap> {
    public TokenSequenceParser() {}

    public CoreMapExpressionExtractor getExpressionExtractor(Env env, Reader r) throws ParseException, TokenSequenceParseException {
        try{
            TokenSequenceParser p = new TokenSequenceParser(r);
            List<SequenceMatchRules.Rule> rules = p.RuleList(env);
            return new CoreMapExpressionExtractor(env, rules);
        }catch(TokenMgrError error){
            throw new TokenSequenceParseException("Parsing failed. Error: " + error);
        }
    }

    public void updateExpressionExtractor(CoreMapExpressionExtractor extractor, Reader r) throws ParseException, TokenSequenceParseException {
        try{
            TokenSequenceParser p = new TokenSequenceParser(r);
            List<SequenceMatchRules.Rule> rules = p.RuleList(extractor.getEnv());
            extractor.appendRules(rules);
        }catch(TokenMgrError error){
            throw new TokenSequenceParseException("Parsing failed. Error: " + error);
        }
    }

	public SequencePattern.PatternExpr parseSequence(Env env, String s) throws ParseException, TokenSequenceParseException {
        try{
            TokenSequenceParser p = new TokenSequenceParser(new StringReader(s));
            return p.SeqRegex(env);
        }catch(TokenMgrError error){

View on GitHub (pinned to 1b7edd19c4)