stanfordnlp/CoreNLP · error · TokenSequenceParseException

Parsing failed. Error:

Error message

Parsing failed. Error: 

What it means

TokenSequenceParser.getExpressionExtractor catches TokenMgrError (the JavaCC lexer's error type, an Error not an Exception) while parsing a rules file and rethrows it as TokenSequenceParseException('Parsing failed. Error: ' + error). It means the rule file contains characters/tokens the TokensRegex lexer cannot recognize.

Solutions

1. Read the wrapped TokenMgrError message for the offending character and line.
2. Re-save the rules file as plain UTF-8/ASCII, replacing smart quotes and em-dashes with ASCII equivalents.
3. Check for unterminated string or regex literals in the rules file.
4. Validate the file against working example rule files shipped with CoreNLP.

Example fix

// before (rules file)
word: “President”   // smart quotes break the lexer
// after
word: "President"     // ASCII quotes

// pre-scan rules file for characters the lexer rejects
String text = new String(Files.readAllBytes(path), StandardCharsets.UTF_8);
for (int i = 0; i < text.length(); i++) {
    char c = text.charAt(i);
    if (c == '\u201C' || c == '\u201D' || c == '\u2018' || c == '\u2019') {
        throw new IllegalStateException("smart quote at offset " + i + " in " + path);
    }
}

try {
    CoreMapExpressionExtractor ext = parser.getExpressionExtractor(env, reader);
} catch (TokenSequenceParseException e) {
    log.error("Rules file lexing failed: " + e.getMessage());
}

When it happens

Trigger: getExpressionExtractor(env, reader) with a Reader over a rule file containing illegal characters, unterminated strings/regex literals, or non-UTF8 bytes that break tokenization.

Common situations: TokenRules files edited with smart quotes or invisible Unicode; wrong encoding (e.g., Latin-1 file read as UTF-8); copied rules with curly braces mismatched at the lexer level.

Related errors

• Not a valid ellipses style
• Not a valid dashes style
• SpanishLexer: Invalid option value in constructor
• Error parsing file:
• Invalid annotation key

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/691a90205322612d. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/ling/tokensregex/parser/TokenSequenceParser.java:26

import edu.stanford.nlp.ling.tokensregex.*;
import edu.stanford.nlp.util.CoreMap;
import edu.stanford.nlp.util.ArrayMap;
import edu.stanford.nlp.util.Pair;
import java.io.Reader;
import java.io.StringReader;
import java.util.*;
import java.lang.RuntimeException;

public class TokenSequenceParser implements SequencePattern.Parser<CoreMap>, TokenSequenceParserConstants {
    public TokenSequenceParser() {}

    public CoreMapExpressionExtractor getExpressionExtractor(Env env, Reader r) throws ParseException, TokenSequenceParseException {
        try{
            TokenSequenceParser p = new TokenSequenceParser(r);
            List<SequenceMatchRules.Rule> rules = p.RuleList(env);
            return new CoreMapExpressionExtractor(env, rules);
        }catch(TokenMgrError error){
            throw new TokenSequenceParseException("Parsing failed. Error: " + error);
        }
    }

    public void updateExpressionExtractor(CoreMapExpressionExtractor extractor, Reader r) throws ParseException, TokenSequenceParseException {
        try{
            TokenSequenceParser p = new TokenSequenceParser(r);
            List<SequenceMatchRules.Rule> rules = p.RuleList(extractor.getEnv());
            extractor.appendRules(rules);
        }catch(TokenMgrError error){
            throw new TokenSequenceParseException("Parsing failed. Error: " + error);
        }
    }

        public SequencePattern.PatternExpr parseSequence(Env env, String s) throws ParseException, TokenSequenceParseException {
        try{
            TokenSequenceParser p = new TokenSequenceParser(new StringReader(s));
            return p.SeqRegex(env);
        }catch(TokenMgrError error){

View on GitHub (pinned to 1b7edd19c4)