{"record":{"id":"0f30ae102a2f2c3d","repo":"stanfordnlp/CoreNLP","slug":"zero-length-token-list-for","errorCode":null,"errorMessage":": Zero length token list for: ","messagePattern":": Zero length token list for: ","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/trees/international/french/FrenchXMLTreeReader.java","lineNumber":208,"sourceCode":"  private List<String> getWordString(String text) {\n    List<String> toks = new ArrayList<>();\n    if(text == null || text.equals(\"\"))\n      toks.add(EMPTY_LEAF);\n    else {\n      //Strip spurious parens\n      if(text.length() > 1)\n        text = text.replaceAll(\"[\\\\(\\\\)]\", \"\");\n\n      //Check for numbers and punctuation\n      String noWhitespaceStr = text.replaceAll(\"\\\\s+\", \"\");\n      if(noWhitespaceStr.matches(\"\\\\d+\") || noWhitespaceStr.matches(\"\\\\p{Punct}+\"))\n        toks.add(noWhitespaceStr);\n      else\n        toks = Arrays.asList(text.split(\"\\\\s+\"));\n    }\n\n    if(toks.size() == 0)\n      throw new RuntimeException(this.getClass().getName() + \": Zero length token list for: \" + text);\n\n    return toks;\n  }\n\n  private Tree getTreeFromXML(Node root) {\n    final Element eRoot = (Element) root;\n\n    if (eRoot.getNodeName().equals(NODE_WORD) &&\n        eRoot.getElementsByTagName(NODE_WORD).getLength() == 0) {\n      String posStr = getPOS(eRoot);\n      posStr = treeNormalizer.normalizeNonterminal(posStr);\n\n      List<String> lemmas = getLemma(eRoot);\n      String morph = getMorph(eRoot);\n      List<String> leafToks = getWordString(eRoot.getTextContent().trim());\n      String subcat = getSubcat(eRoot);\n\n      if (lemmas != null && lemmas.size() != leafToks.size()) {","sourceCodeStart":190,"sourceCodeEnd":226,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/trees/international/french/FrenchXMLTreeReader.java#L190-L226","documentation":"FrenchXMLTreeReader's token-list builder splits the <text> content of an XML element on whitespace and requires at least one token; if the text is null/empty or collapses to nothing after filtering, it throws a RuntimeException naming the offending text. This guards downstream lemma/leaf reconstruction, which would otherwise produce trees with missing leaf nodes.","triggerScenarios":"Reading a French XML corpus tree whose word/text element is empty, whitespace-only after stripping, or null — encountered when getLemma/leafToks calls getWordString on such an element.","commonSituations":"Corpus files with empty <w> elements or malformed records; XML entities that strip to nothing after noWhitespace filtering; truncated or partially downloaded corpus files; preprocessing scripts that blanked text fields.","solutions":["Inspect the offending file and fill or remove the empty text element (the exception message includes the offending text, usually blank/null)","Filter out malformed records before reading, e.g. validate XML elements have non-empty text with a pre-pass","Strip junk characters (soft hyphens, NBSP) that collapse to empty after split so the element retains real tokens","Wrap the read in try/catch for RuntimeException and skip/log the bad tree to continue corpus loading"],"exampleFix":"// before\nString text = element.getTextContent(); // may be empty\nTree t = reader.read(); // throws \"Zero length token list\"\n\n// after\nString text = element.getTextContent();\nif (text != null && !text.trim().isEmpty()) {\n  Tree t = reader.read();\n} else {\n  log.warn(\"Skipping empty text element\");\n}","handlingStrategy":"try-catch","validationCode":"// pre-validate XML elements before feeding the reader\nString text = element.getTextContent();\nif (text == null || text.trim().isEmpty())\n  throw new SkipRecordException(\"empty text element at \" + element.getTagName());","typeGuard":"static boolean hasTokenizableText(Element e) {\n  String t = e.getTextContent();\n  return t != null && !t.trim().isEmpty();\n}","tryCatchPattern":"try {\n  Tree t = reader.read();\n  trees.add(t);\n} catch (RuntimeException e) {\n  if (e.getMessage().contains(\"Zero length token list\")) {\n    log.warn(\"Skipping malformed record: \" + e.getMessage());\n  } else {\n    throw e;\n  }\n}","preventionTips":["Validate corpus XML for empty or whitespace-only text elements before batch reading","Normalize exotic whitespace (NBSP, soft hyphen) that survives the split and yields zero tokens","Scan-corpus pre-pass: split on \\\\s+ and count tokens, dropping empties with a logged record ID","Keep checksums on corpus files to detect truncated downloads"],"tags":["java","nlp","xml","corpus"],"backgroundTag":"empty-required-field","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}