stanfordnlp/CoreNLP · error · RuntimeException

: Zero length token list for:

Error message

: Zero length token list for: 

What it means

FrenchXMLTreeReader's token-list builder splits the <text> content of an XML element on whitespace and requires at least one token; if the text is null/empty or collapses to nothing after filtering, it throws a RuntimeException naming the offending text. This guards downstream lemma/leaf reconstruction, which would otherwise produce trees with missing leaf nodes.

Solutions

1. Inspect the offending file and fill or remove the empty text element (the exception message includes the offending text, usually blank/null)
2. Filter out malformed records before reading, e.g. validate XML elements have non-empty text with a pre-pass
3. Strip junk characters (soft hyphens, NBSP) that collapse to empty after split so the element retains real tokens
4. Wrap the read in try/catch for RuntimeException and skip/log the bad tree to continue corpus loading

Example fix

// before
String text = element.getTextContent(); // may be empty
Tree t = reader.read(); // throws "Zero length token list"

// after
String text = element.getTextContent();
if (text != null && !text.trim().isEmpty()) {
  Tree t = reader.read();
} else {
  log.warn("Skipping empty text element");
}

// pre-validate XML elements before feeding the reader
String text = element.getTextContent();
if (text == null || text.trim().isEmpty())
  throw new SkipRecordException("empty text element at " + element.getTagName());

static boolean hasTokenizableText(Element e) {
  String t = e.getTextContent();
  return t != null && !t.trim().isEmpty();
}

try {
  Tree t = reader.read();
  trees.add(t);
} catch (RuntimeException e) {
  if (e.getMessage().contains("Zero length token list")) {
    log.warn("Skipping malformed record: " + e.getMessage());
  } else {
    throw e;
  }
}

When it happens

Trigger: Reading a French XML corpus tree whose word/text element is empty, whitespace-only after stripping, or null — encountered when getLemma/leafToks calls getWordString on such an element.

Common situations: Corpus files with empty <w> elements or malformed records; XML entities that strip to nothing after noWhitespace filtering; truncated or partially downloaded corpus files; preprocessing scripts that blanked text fields.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• TIMEX3 should only contain text
• unexpected content
• unexpected element
• after W derivative, index() != x.length()
• An error occurred while testing the tagger.

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/0f30ae102a2f2c3d. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/trees/international/french/FrenchXMLTreeReader.java:208

  private List<String> getWordString(String text) {
    List<String> toks = new ArrayList<>();
    if(text == null || text.equals(""))
      toks.add(EMPTY_LEAF);
    else {
      //Strip spurious parens
      if(text.length() > 1)
        text = text.replaceAll("[\\(\\)]", "");

      //Check for numbers and punctuation
      String noWhitespaceStr = text.replaceAll("\\s+", "");
      if(noWhitespaceStr.matches("\\d+") || noWhitespaceStr.matches("\\p{Punct}+"))
        toks.add(noWhitespaceStr);
      else
        toks = Arrays.asList(text.split("\\s+"));
    }

    if(toks.size() == 0)
      throw new RuntimeException(this.getClass().getName() + ": Zero length token list for: " + text);

    return toks;
  }

  private Tree getTreeFromXML(Node root) {
    final Element eRoot = (Element) root;

    if (eRoot.getNodeName().equals(NODE_WORD) &&
        eRoot.getElementsByTagName(NODE_WORD).getLength() == 0) {
      String posStr = getPOS(eRoot);
      posStr = treeNormalizer.normalizeNonterminal(posStr);

      List<String> lemmas = getLemma(eRoot);
      String morph = getMorph(eRoot);
      List<String> leafToks = getWordString(eRoot.getTextContent().trim());
      String subcat = getSubcat(eRoot);

      if (lemmas != null && lemmas.size() != leafToks.size()) {

View on GitHub (pinned to 1b7edd19c4)