stanfordnlp/CoreNLP · error · IllegalStateException

Unknown special lemma:

Error message

Unknown special lemma:

What it means

In keyphraseSpans, special lemmas (like "'s") are handled by a switch with a default that throws IllegalStateException('Unknown special lemma: ' + lemma) when a lemma in the special-lemma set has no case in the FSA. This happens when the token set the code treats as special and the switch's cases drift apart.

Solutions

  1. Inspect sentence.lemmas() to find the offending lemma
  2. Upgrade CoreNLP — newer versions add cases for more special lemmas
  3. Normalize/strip unusual punctuation before running keyphrase extraction
  4. Catch the IllegalStateException and skip that sentence

Example fix

// before
List<String> kps = sentence.algorithms().keyphrases();
// after
List<String> kps;
try {
  kps = sentence.algorithms().keyphrases();
} catch (IllegalStateException e) {
  kps = Collections.emptyList(); // unknown special lemma in this sentence
}
Defensive patterns

Strategy: try-catch

Validate before calling

// pre-scan for lemmas the FSA may not know
boolean lemmasKnown = sentence.lemmas().stream()
  .allMatch(l -> !l.startsWith("'") || l.equals("'s"));

Try / catch

try {
  List<String> kps = sentence.algorithms().keyphrases();
} catch (IllegalStateException e) {
  if (e.getMessage().startsWith("Unknown special lemma")) {
    kps = Collections.emptyList();
  } else { throw e; }
}

Prevention

When it happens

Trigger: Calling keyphraseSpans()/keyphrases() on a sentence containing a token whose lemma is classified as 'special' but lacks a switch case — e.g. unusual punctuation or contractions beyond those handled ('s, and, etc. depending on version).

Common situations: Text with uncommon contractions or symbols; using a lemmatizer whose output forms differ from what SentenceAlgorithms expects (e.g. custom lemmatizer or different language model).

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/f67eeeae084011bb. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/simple/SentenceAlgorithms.java:177

          //       update the transition matrix.
          updateExpectation.accept(coarseTag);
          inLookahead = false;
        } else if (expectNextLemma.contains(lemma)) {
          // Case: we hit a valid word. Do something special.
          switch (lemma) {
            case "of":
              // These prepositions are valid to subsume into a noun phrase.
              // Update the transition matrix, and mark this as conditionally ok.
              updateExpectation.accept('I');
              inLookahead = true;
              break;
            case "'s":
              // Possessives often denote a longer compound phrase
              updateExpectation.accept('Z');
              inLookahead = true;
              break;
            default:
              throw new IllegalStateException("Unknown special lemma: " + lemma);
          }
        } else {
          // Case: We have transitioned to an 'invalid' state, and therefore the span should end.
          if (inLookahead) {
            // If we were in a lookahead token, ignore the last token (as per the lookahead definition)
            spans.add(Span.fromValues(spanBegin, i - 1));
          } else {
            // Otherwise, add the span
            spans.add(Span.fromValues(spanBegin, i));
          }
          // We may also have started a new span.
          // Check to see if we have started a new span.
          if (coarseTag == 'N' || coarseTag == 'V' || coarseTag == 'J' || coarseTag == 'X' || coarseTag == 'G') {
            spanBegin = i;
            updateExpectation.accept(coarseTag);
          } else {
            spanBegin = -1;
          }

View on GitHub (pinned to 1b7edd19c4)