stanfordnlp/CoreNLP · error · RuntimeException

Expected token to be either Word or String.

Error message

Expected token to be either Word or String.

What it means

WordToSentenceProcessor.getString extracts the text of a token for sentence-break matching. Tokens must be Word, String, or CoreMap instances; anything else triggers this RuntimeException. It means a non-standard object ended up in the token list.

Solutions

1. Convert tokens to CoreLabel or Word objects before sentence splitting
2. If a custom Label is used, make it implement HasWord and return a Word, or extract its text into Strings
3. Use CoreMap (CoreLabel) tokens as the annotator pipeline does
4. Check where the list is built and normalize token types there

Example fix

// before
List<Object> tokens = myCustomTokens; // contains MyLabel
// after
List<CoreLabel> tokens = myCustomTokens.stream()
    .map(t -> CoreLabel.wordFromString(t.toString()))
    .collect(Collectors.toList());

boolean allSupported(List<?> tokens) {
  return tokens.stream().allMatch(t -> t instanceof Word || t instanceof String || t instanceof CoreMap);
}

String tokenText(Object o) {
  if (o instanceof CoreMap) return ((CoreMap) o).get(CoreAnnotations.TextAnnotation.class);
  if (o instanceof Word) return ((Word) o).word();
  if (o instanceof String) return (String) o;
  return o == null ? null : o.toString();
}

try {
  List<List<Word>> sents = processor.process(tokens);
} catch (RuntimeException e) {
  if (e.getMessage().contains("Expected token")) tokens = normalize(tokens);
  else throw e;
}

When it happens

Trigger: Passing a List<Object> to WordToSentenceProcessor (e.g. via the wordsToSentences annotator path or direct API) containing objects that are not Word, String, or CoreMap — e.g. custom Label implementations or HasWord wrappers other than Word.

Common situations: Building token lists manually with a custom Label class; mixing token types after deserialization; feeding List<HasWord> items that are raw Labels lacking word().

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• addFeature was called with a features object that is…
• ArabicLexer: the invertible option requires a…
• Attempting to remove features based on weight from a…
• Cannot cast " + classname + " into " + type.getName()
• Cannot process a single key/value from annotation as it is…

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/f08503c782d67a7c. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/WordToSentenceProcessor.java:178

      String originalText = ((CoreMap) o).get(CoreAnnotations.OriginalTextAnnotation.class);
      return (forcedEndValue != null && forcedEndValue) ||
          (originalText != null && originalText.equals("\u2029"));
    } else {
      return false;
    }
  }

  @SuppressWarnings("OverlyStrongTypeCast")
  private static String getString(Object o) {
    if (o instanceof HasWord) {
      HasWord h = (HasWord) o;
      return h.word();
    } else if (o instanceof String) {
      return (String) o;
    } else if (o instanceof CoreMap) {
      return ((CoreMap) o).get(CoreAnnotations.TextAnnotation.class);
    } else {
      throw new RuntimeException("Expected token to be either Word or String.");
    }
  }

  @SuppressWarnings("Convert2streamapi")
  private static boolean matches(List<Pattern> patterns, String word) {
    for (Pattern p: patterns) {
      Matcher m = p.matcher(word);
      if (m.matches()) {
        return true;
      }
    }
    return false;
  }

  private boolean matchesXmlBreakElementToDiscard(String word) {
    return matches(xmlBreakElementsToDiscard, word);
  }

View on GitHub (pinned to 1b7edd19c4)