stanfordnlp/CoreNLP · error · IllegalArgumentException

Input word not tagged

Error message

Input word not tagged

What it means

In ShiftReduceParser.initialStateFromTaggedSentence, after extracting the tag from a HasTag word, if the resulting tag is null the word was nominally tagged but has no actual tag value. The parser cannot build its initial state without tags for every token, so it throws IllegalArgumentException("Input word not tagged").

Solutions

  1. Ensure the POS tagger runs over the full sentence and assigns a tag to every token before calling the parser
  2. Check each token's tag for null before invoking and fix upstream tagging code
  3. If constructing tokens manually, call setTag on every word

Example fix

// before
CoreLabel cl = new CoreLabel();
cl.setWord("dog");
sentence.add(cl); // tag is null

// after
CoreLabel cl = new CoreLabel();
cl.setWord("dog");
cl.setTag("NN");
sentence.add(cl);
Defensive patterns

Strategy: validation

Validate before calling

boolean allTagged = sentence.stream().allMatch(w -> ((HasTag) w).tag() != null);
if (!allTagged) { throw new IllegalArgumentException("Some words have null tags"); }

Type guard

boolean hasTag(HasWord hw) {
  return hw instanceof HasTag && ((HasTag) hw).tag() != null && !((HasTag) hw).tag().isEmpty();
}

Try / catch

try {
  state = parser.initialStateFromTaggedSentence(sentence);
} catch (IllegalArgumentException e) {
  if (e.getMessage().equals("Input word not tagged")) {
    sentence = retagger.tagSentence(rawTokens);
    state = parser.initialStateFromTaggedSentence(sentence);
  } else throw e;
}

Prevention

When it happens

Trigger: Passing HasTag/CoreLabel tokens whose tag field was never set (e.g. CoreLabels straight from a tokenizer, or TaggedWord created without a tag / setTag(null)).

Common situations: Tokenizer output (CoreLabel without tag) passed where tagged output is expected; a tagger skipping low-confidence tokens leaving tags null; words like unknown/punctuation added after tagging.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/a7fc1489930d946f. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/parser/shiftreduce/ShiftReduceParser.java:251

      HasWord hw = words.get(index);

      CoreLabel wordLabel;
      String tag;
      if (hw instanceof CoreLabel) {
        wordLabel = (CoreLabel) hw;
        tag = wordLabel.tag();
      } else {
        wordLabel = new CoreLabel();
        wordLabel.setValue(hw.word());
        wordLabel.setWord(hw.word());
        if (!(hw instanceof HasTag)) {
          throw new IllegalArgumentException("Expected tagged words");
        }
        tag = ((HasTag) hw).tag();
        wordLabel.setTag(tag);
      }
      if (tag == null) {
        throw new IllegalArgumentException("Input word not tagged");
      }
      CoreLabel tagLabel = new CoreLabel();
      tagLabel.setValue(tag);

      // Index from 1.  Tools downstream from the parser expect that
      // Internally this parser uses the index, so we have to
      // overwrite incorrect indices if the label is already indexed
      wordLabel.setIndex(index + 1);
      tagLabel.setIndex(index + 1);

      LabeledScoredTreeNode wordNode = new LabeledScoredTreeNode(wordLabel);
      LabeledScoredTreeNode tagNode = new LabeledScoredTreeNode(tagLabel);
      tagNode.addChild(wordNode);

      // TODO: can we get away with not setting these on the wordLabel?
      wordLabel.set(TreeCoreAnnotations.HeadWordLabelAnnotation.class, wordLabel);
      wordLabel.set(TreeCoreAnnotations.HeadTagLabelAnnotation.class, tagLabel);
      tagLabel.set(TreeCoreAnnotations.HeadWordLabelAnnotation.class, wordLabel);

View on GitHub (pinned to 1b7edd19c4)