{"record":{"id":"f08503c782d67a7c","repo":"stanfordnlp/CoreNLP","slug":"expected-token-to-be-either-word-or-string","errorCode":null,"errorMessage":"Expected token to be either Word or String.","messagePattern":"Expected token to be either Word or String\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/process/WordToSentenceProcessor.java","lineNumber":178,"sourceCode":"      String originalText = ((CoreMap) o).get(CoreAnnotations.OriginalTextAnnotation.class);\n      return (forcedEndValue != null && forcedEndValue) ||\n          (originalText != null && originalText.equals(\"\\u2029\"));\n    } else {\n      return false;\n    }\n  }\n\n  @SuppressWarnings(\"OverlyStrongTypeCast\")\n  private static String getString(Object o) {\n    if (o instanceof HasWord) {\n      HasWord h = (HasWord) o;\n      return h.word();\n    } else if (o instanceof String) {\n      return (String) o;\n    } else if (o instanceof CoreMap) {\n      return ((CoreMap) o).get(CoreAnnotations.TextAnnotation.class);\n    } else {\n      throw new RuntimeException(\"Expected token to be either Word or String.\");\n    }\n  }\n\n  @SuppressWarnings(\"Convert2streamapi\")\n  private static boolean matches(List<Pattern> patterns, String word) {\n    for (Pattern p: patterns) {\n      Matcher m = p.matcher(word);\n      if (m.matches()) {\n        return true;\n      }\n    }\n    return false;\n  }\n\n  private boolean matchesXmlBreakElementToDiscard(String word) {\n    return matches(xmlBreakElementsToDiscard, word);\n  }\n","sourceCodeStart":160,"sourceCodeEnd":196,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/process/WordToSentenceProcessor.java#L160-L196","documentation":"WordToSentenceProcessor.getString extracts the text of a token for sentence-break matching. Tokens must be Word, String, or CoreMap instances; anything else triggers this RuntimeException. It means a non-standard object ended up in the token list.","triggerScenarios":"Passing a List<Object> to WordToSentenceProcessor (e.g. via the wordsToSentences annotator path or direct API) containing objects that are not Word, String, or CoreMap — e.g. custom Label implementations or HasWord wrappers other than Word.","commonSituations":"Building token lists manually with a custom Label class; mixing token types after deserialization; feeding List<HasWord> items that are raw Labels lacking word().","solutions":["Convert tokens to CoreLabel or Word objects before sentence splitting","If a custom Label is used, make it implement HasWord and return a Word, or extract its text into Strings","Use CoreMap (CoreLabel) tokens as the annotator pipeline does","Check where the list is built and normalize token types there"],"exampleFix":"// before\nList<Object> tokens = myCustomTokens; // contains MyLabel\n// after\nList<CoreLabel> tokens = myCustomTokens.stream()\n    .map(t -> CoreLabel.wordFromString(t.toString()))\n    .collect(Collectors.toList());","handlingStrategy":"type-guard","validationCode":"boolean allSupported(List<?> tokens) {\n  return tokens.stream().allMatch(t -> t instanceof Word || t instanceof String || t instanceof CoreMap);\n}","typeGuard":"String tokenText(Object o) {\n  if (o instanceof CoreMap) return ((CoreMap) o).get(CoreAnnotations.TextAnnotation.class);\n  if (o instanceof Word) return ((Word) o).word();\n  if (o instanceof String) return (String) o;\n  return o == null ? null : o.toString();\n}","tryCatchPattern":"try {\n  List<List<Word>> sents = processor.process(tokens);\n} catch (RuntimeException e) {\n  if (e.getMessage().contains(\"Expected token\")) tokens = normalize(tokens);\n  else throw e;\n}","preventionTips":["Build token lists only from CoreLabel/Word/String","Convert custom Label types before sentence splitting","Keep pipeline-produced annotations for the ssplit annotator"],"tags":["java","type-mismatch","tokenization"],"backgroundTag":"type-mismatch","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}