stanfordnlp/CoreNLP · error · IllegalStateException

Cannot update expected next token for POS tag:

Error message

Cannot update expected next token for POS tag:

What it means

SentenceAlgorithms.keyphraseSpans runs a finite-state machine over POS tags; the updateExpectation lambda throws IllegalStateException when the coarse POS tag does not match any of the states it knows how to transition from. A tag outside the recognized families (noun/N, X, adjective/J, verb/V, preposition/P, etc.) breaks the FSA.

Solutions

  1. Use the default English Penn-Treebank POS tagger before computing keyphrases
  2. Inspect the tags with sentence.posTags() and identify the offending tag
  3. Extend/patch the FSA in SentenceAlgorithms to handle the unexpected coarse tag
  4. Avoid keyphrases() for non-English tagsets; implement language-specific extraction

Example fix

// before
List<Span> spans = sentence.algorithms().keyphraseSpans();
// after
if (!sentence.posTags().stream().allMatch(t -> t.matches("[NNP?S|JJ|VB.*|IN|PRP|DT|POS|X|,|\\.].*"))) {
  throw new IllegalArgumentException("Unrecognized tagset for keyphrase extraction");
}
List<Span> spans = sentence.algorithms().keyphraseSpans();
Defensive patterns

Strategy: try-catch

Validate before calling

// check for unexpected coarse tags first
Set<Character> known = Set.of('N','X','J','V','P','M','Z');
boolean tagsOk = sentence.posTags().stream()
  .allMatch(t -> known.contains(t.charAt(0)));

Try / catch

try {
  List<Span> spans = sentence.algorithms().keyphraseSpans();
} catch (IllegalStateException e) {
  if (e.getMessage().startsWith("Cannot update expected next token")) {
    spans = Collections.emptyList();
  } else { throw e; }
}

Prevention

When it happens

Trigger: Calling sentence.algorithms().keyphrases() or keyphraseSpans() on a sentence POS-tagged with a tagger whose tags don't map to expected coarse categories — e.g. a non-Penn-Treebank tagger, foreign-language models, or corrupted tag output.

Common situations: Running keyphrases() on non-English text tagged with a language-specific tagset; using a custom POS model with novel tags; feeding pre-tagged tokens with tags the FSA never anticipated.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/662b01407866c135. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/simple/SentenceAlgorithms.java:125

        expectNextTag.add('J');
        expectNextLemma.clear();
      } else if (coarseTag == 'V') {
        expectNextTag.clear();
        expectNextTag.add('V');
        expectNextLemma.clear();
      } else if (coarseTag == 'Z') {
        expectNextTag.clear();
        expectNextTag.add('J');
        expectNextTag.add('N');
        expectNextLemma.clear();
      } else if (coarseTag == 'I') {
        expectNextTag.clear();
        expectNextTag.add('N');
        expectNextTag.add('X');
        expectNextTag.add('J');
        expectNextLemma.clear();
      } else {
        throw new IllegalStateException("Cannot update expected next token for POS tag: " + coarseTag);
      }
    };

    // Run the FSA:
    for (int i = 0; i < sentence.length(); ++i) {
      // Get some variables
      String tag = sentence.posTag(i);
      char coarseTag = Character.toUpperCase(tag.charAt(0));
      String lemma = sentence.lemma(i).toLowerCase();
      // Tweak the tag
      if (coarseTag == 'V' && lemma.equals("be")) {
        coarseTag = 'B';
      } else if (tag.startsWith("NNP")) {
        coarseTag = 'X';
      } else if (tag.startsWith("POS")) {
        coarseTag = 'Z';
      }
      // (don't collapse 'ing' nouns)

View on GitHub (pinned to 1b7edd19c4)