{"record":{"id":"f04e5bee128cbb35","repo":"stanfordnlp/CoreNLP","slug":"found-line-with-label-line-but-no-tokens-t","errorCode":null,"errorMessage":"Found line with label \" + line + \" but no tokens to associate with that line","messagePattern":"Found line with label \" \\+ line \\+ \" but no tokens to associate with that line","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/sentiment/BuildBinarizedDataset.java","lineNumber":66,"sourceCode":"    if (tree.isLeaf()) {\n      return;\n    }\n\n    for (Tree child : tree.children()) {\n      setPredictedLabels(child);\n    }\n\n    tree.label().setValue(Integer.toString(RNNCoreAnnotations.getPredictedClass(tree)));\n  }\n\n  public static void extractLabels(Map<Pair<Integer, Integer>, String> spanToLabels, List<HasWord> tokens, String line) {\n    String[] pieces = line.trim().split(\"\\\\s+\");\n    if (pieces.length == 0) {\n      return;\n    }\n    if (pieces.length == 1) {\n      String error = \"Found line with label \" + line + \" but no tokens to associate with that line\";\n      throw new RuntimeException(error);\n    }\n\n    //TODO: BUG: The pieces are tokenized differently than the splitting, e.g., on possessive markers as in \"actors' expenses\"\n    for (int i = 0; i < tokens.size() - pieces.length + 2; ++i) {\n      boolean found = true;\n      for (int j = 1; j < pieces.length; ++j) {\n        if (!tokens.get(i + j - 1).word().equals(pieces[j])) {\n          found = false;\n          break;\n        }\n      }\n      if (found) {\n        spanToLabels.put(new Pair<>(i, i + pieces.length - 1), pieces[0]);\n      }\n    }\n  }\n\n  public static boolean setSpanLabel(Tree tree, Pair<Integer, Integer> span, String value) {","sourceCodeStart":48,"sourceCodeEnd":84,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/sentiment/BuildBinarizedDataset.java#L48-L84","documentation":"BuildBinarizedDataset.extractLabels parses each input line as a sentiment label followed by the phrase tokens. A line containing only one whitespace-delimited piece means there is a label but no tokens, so it throws RuntimeException indicating the input file is malformed at that line.","triggerScenarios":"A line in the input dataset (e.g. Stanford Sentiment Treebank-style label+phrase file) contains a bare label like \"1\" with no following phrase tokens.","commonSituations":"Blank-ish lines containing only whitespace plus a number; truncated files; copy/paste errors when preparing the training data; files with a trailing label-only line.","solutions":["Open the input file and fix or delete the offending label-only line (file/line is named in the message)","Preprocess the file to skip lines with fewer than 2 tokens","Regenerate or re-download the dataset from a trusted source"],"exampleFix":"// before (bad line in input file)\n1\n// after\n1 the actors ' expenses","handlingStrategy":"validation","validationCode":"List<String> bad = Files.readAllLines(input).stream()\n  .filter(l -> !l.trim().isEmpty())\n  .filter(l -> l.trim().split(\"\\\\s+\").length < 2)\n  .collect(toList());\nif (!bad.isEmpty()) throw new IllegalArgumentException(\"Label-only lines: \" + bad);","typeGuard":"static boolean hasTokens(String line) { return line.trim().split(\"\\\\s+\").length >= 2; }","tryCatchPattern":"try {\n  BuildBinarizedDataset.main(args);\n} catch (RuntimeException e) {\n  log.error(\"Malformed dataset line: \" + e.getMessage());\n}","preventionTips":["Pre-scan dataset files for label-only or blank lines","Regenerate datasets with a fixed writer instead of manual editing","Keep a checksum of known-good dataset files"],"tags":["dataset","parsing","sentiment"],"backgroundTag":"invalid-argument-format","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}