{"record":{"id":"a7fc1489930d946f","repo":"stanfordnlp/CoreNLP","slug":"input-word-not-tagged","errorCode":null,"errorMessage":"Input word not tagged","messagePattern":"Input word not tagged","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/parser/shiftreduce/ShiftReduceParser.java","lineNumber":251,"sourceCode":"      HasWord hw = words.get(index);\n\n      CoreLabel wordLabel;\n      String tag;\n      if (hw instanceof CoreLabel) {\n        wordLabel = (CoreLabel) hw;\n        tag = wordLabel.tag();\n      } else {\n        wordLabel = new CoreLabel();\n        wordLabel.setValue(hw.word());\n        wordLabel.setWord(hw.word());\n        if (!(hw instanceof HasTag)) {\n          throw new IllegalArgumentException(\"Expected tagged words\");\n        }\n        tag = ((HasTag) hw).tag();\n        wordLabel.setTag(tag);\n      }\n      if (tag == null) {\n        throw new IllegalArgumentException(\"Input word not tagged\");\n      }\n      CoreLabel tagLabel = new CoreLabel();\n      tagLabel.setValue(tag);\n\n      // Index from 1.  Tools downstream from the parser expect that\n      // Internally this parser uses the index, so we have to\n      // overwrite incorrect indices if the label is already indexed\n      wordLabel.setIndex(index + 1);\n      tagLabel.setIndex(index + 1);\n\n      LabeledScoredTreeNode wordNode = new LabeledScoredTreeNode(wordLabel);\n      LabeledScoredTreeNode tagNode = new LabeledScoredTreeNode(tagLabel);\n      tagNode.addChild(wordNode);\n\n      // TODO: can we get away with not setting these on the wordLabel?\n      wordLabel.set(TreeCoreAnnotations.HeadWordLabelAnnotation.class, wordLabel);\n      wordLabel.set(TreeCoreAnnotations.HeadTagLabelAnnotation.class, tagLabel);\n      tagLabel.set(TreeCoreAnnotations.HeadWordLabelAnnotation.class, wordLabel);","sourceCodeStart":233,"sourceCodeEnd":269,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/parser/shiftreduce/ShiftReduceParser.java#L233-L269","documentation":"In ShiftReduceParser.initialStateFromTaggedSentence, after extracting the tag from a HasTag word, if the resulting tag is null the word was nominally tagged but has no actual tag value. The parser cannot build its initial state without tags for every token, so it throws IllegalArgumentException(\"Input word not tagged\").","triggerScenarios":"Passing HasTag/CoreLabel tokens whose tag field was never set (e.g. CoreLabels straight from a tokenizer, or TaggedWord created without a tag / setTag(null)).","commonSituations":"Tokenizer output (CoreLabel without tag) passed where tagged output is expected; a tagger skipping low-confidence tokens leaving tags null; words like unknown/punctuation added after tagging.","solutions":["Ensure the POS tagger runs over the full sentence and assigns a tag to every token before calling the parser","Check each token's tag for null before invoking and fix upstream tagging code","If constructing tokens manually, call setTag on every word"],"exampleFix":"// before\nCoreLabel cl = new CoreLabel();\ncl.setWord(\"dog\");\nsentence.add(cl); // tag is null\n\n// after\nCoreLabel cl = new CoreLabel();\ncl.setWord(\"dog\");\ncl.setTag(\"NN\");\nsentence.add(cl);","handlingStrategy":"validation","validationCode":"boolean allTagged = sentence.stream().allMatch(w -> ((HasTag) w).tag() != null);\nif (!allTagged) { throw new IllegalArgumentException(\"Some words have null tags\"); }","typeGuard":"boolean hasTag(HasWord hw) {\n  return hw instanceof HasTag && ((HasTag) hw).tag() != null && !((HasTag) hw).tag().isEmpty();\n}","tryCatchPattern":"try {\n  state = parser.initialStateFromTaggedSentence(sentence);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().equals(\"Input word not tagged\")) {\n    sentence = retagger.tagSentence(rawTokens);\n    state = parser.initialStateFromTaggedSentence(sentence);\n  } else throw e;\n}","preventionTips":["Run the tagger over the complete sentence including punctuation","Never append tokens after tagging","Null-check tags as a pipeline invariant"],"tags":["parser","pos-tags","input-validation","null"],"backgroundTag":"empty-required-field","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}