{"record":{"id":"662b01407866c135","repo":"stanfordnlp/CoreNLP","slug":"cannot-update-expected-next-token-for-pos-tag","errorCode":null,"errorMessage":"Cannot update expected next token for POS tag: ","messagePattern":"Cannot update expected next token for POS tag: ","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/simple/SentenceAlgorithms.java","lineNumber":125,"sourceCode":"        expectNextTag.add('J');\n        expectNextLemma.clear();\n      } else if (coarseTag == 'V') {\n        expectNextTag.clear();\n        expectNextTag.add('V');\n        expectNextLemma.clear();\n      } else if (coarseTag == 'Z') {\n        expectNextTag.clear();\n        expectNextTag.add('J');\n        expectNextTag.add('N');\n        expectNextLemma.clear();\n      } else if (coarseTag == 'I') {\n        expectNextTag.clear();\n        expectNextTag.add('N');\n        expectNextTag.add('X');\n        expectNextTag.add('J');\n        expectNextLemma.clear();\n      } else {\n        throw new IllegalStateException(\"Cannot update expected next token for POS tag: \" + coarseTag);\n      }\n    };\n\n    // Run the FSA:\n    for (int i = 0; i < sentence.length(); ++i) {\n      // Get some variables\n      String tag = sentence.posTag(i);\n      char coarseTag = Character.toUpperCase(tag.charAt(0));\n      String lemma = sentence.lemma(i).toLowerCase();\n      // Tweak the tag\n      if (coarseTag == 'V' && lemma.equals(\"be\")) {\n        coarseTag = 'B';\n      } else if (tag.startsWith(\"NNP\")) {\n        coarseTag = 'X';\n      } else if (tag.startsWith(\"POS\")) {\n        coarseTag = 'Z';\n      }\n      // (don't collapse 'ing' nouns)","sourceCodeStart":107,"sourceCodeEnd":143,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/simple/SentenceAlgorithms.java#L107-L143","documentation":"SentenceAlgorithms.keyphraseSpans runs a finite-state machine over POS tags; the updateExpectation lambda throws IllegalStateException when the coarse POS tag does not match any of the states it knows how to transition from. A tag outside the recognized families (noun/N, X, adjective/J, verb/V, preposition/P, etc.) breaks the FSA.","triggerScenarios":"Calling sentence.algorithms().keyphrases() or keyphraseSpans() on a sentence POS-tagged with a tagger whose tags don't map to expected coarse categories — e.g. a non-Penn-Treebank tagger, foreign-language models, or corrupted tag output.","commonSituations":"Running keyphrases() on non-English text tagged with a language-specific tagset; using a custom POS model with novel tags; feeding pre-tagged tokens with tags the FSA never anticipated.","solutions":["Use the default English Penn-Treebank POS tagger before computing keyphrases","Inspect the tags with sentence.posTags() and identify the offending tag","Extend/patch the FSA in SentenceAlgorithms to handle the unexpected coarse tag","Avoid keyphrases() for non-English tagsets; implement language-specific extraction"],"exampleFix":"// before\nList<Span> spans = sentence.algorithms().keyphraseSpans();\n// after\nif (!sentence.posTags().stream().allMatch(t -> t.matches(\"[NNP?S|JJ|VB.*|IN|PRP|DT|POS|X|,|\\\\.].*\"))) {\n  throw new IllegalArgumentException(\"Unrecognized tagset for keyphrase extraction\");\n}\nList<Span> spans = sentence.algorithms().keyphraseSpans();","handlingStrategy":"try-catch","validationCode":"// check for unexpected coarse tags first\nSet<Character> known = Set.of('N','X','J','V','P','M','Z');\nboolean tagsOk = sentence.posTags().stream()\n  .allMatch(t -> known.contains(t.charAt(0)));","typeGuard":null,"tryCatchPattern":"try {\n  List<Span> spans = sentence.algorithms().keyphraseSpans();\n} catch (IllegalStateException e) {\n  if (e.getMessage().startsWith(\"Cannot update expected next token\")) {\n    spans = Collections.emptyList();\n  } else { throw e; }\n}","preventionTips":["Use the standard English PTB POS tagger before keyphrase extraction","Pre-scan posTags() for non-English or custom tagsets","Skip keyphrases() for sentences tagged with unsupported models"],"tags":["corenlp","keyphrase","pos-tags"],"backgroundTag":"invalid-state-transition","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}