{"record":{"id":"c77a4857357eb57c","repo":"stanfordnlp/CoreNLP","slug":"cannot-compute-precision-and-recall-on-unlabelled","errorCode":null,"errorMessage":"Cannot compute precision and recall on unlabelled dataset. Offending datum: ${datum}","messagePattern":"Cannot compute precision and recall on unlabelled dataset\\. Offending datum: (.+?)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/classify/Classifier.java","lineNumber":51,"sourceCode":"   *\n   * @param testData The dataset to evaluate the classifier on.\n   * @param targetLabel The target label (e.g., for relation extraction, this is the relation we're interested in).\n   * @return A pair of the precision (first) and recall (second) of the classifier on the target label.\n   */\n  public default Pair<Double, Double> evaluatePrecisionAndRecall(GeneralDataset<L, F> testData, L targetLabel) {\n    if (targetLabel == null) {\n      throw new IllegalArgumentException(\"Must supply a target label to compute precision and recall against\");\n    }\n    // Variables to count\n    int numCorrectAndTarget = 0;\n    int numTargetGuess = 0;\n    int numTargetGold = 0;\n    // Iterate over dataset\n    for (RVFDatum<L, F> datum : testData) {\n      // Get the gold label\n      L label = datum.label();\n      if (label == null) {\n        throw new IllegalArgumentException(\"Cannot compute precision and recall on unlabelled dataset. Offending datum: \" + datum);\n      }\n      // Get the guess label\n      L guess = classOf(datum);\n      // Compute statistics on datum\n      if (label.equals(targetLabel)) {\n        numTargetGold += 1;\n      }\n      if (guess.equals(targetLabel)) {\n        numTargetGuess += 1;\n        if (guess.equals(label)) {\n          numCorrectAndTarget += 1;\n        }\n      }\n    }\n    // Aggregate statistics\n    double precision = numTargetGuess == 0 ? 0.0 : ((double) numCorrectAndTarget) / ((double) numTargetGuess);\n    double recall = numTargetGold == 0 ? 1.0 : ((double) numCorrectAndTarget) / ((double) numTargetGold);\n    return Pair.makePair(precision, recall);","sourceCodeStart":33,"sourceCodeEnd":69,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/classify/Classifier.java#L33-L69","documentation":"While iterating the test dataset to compute precision/recall against a target label, the library requires every datum to carry a gold label. If datum.label() returns null for any RVFDatum, it throws an IllegalArgumentException identifying the offending datum, because unlabeled examples cannot contribute to precision/recall counts.","triggerScenarios":"Calling evaluatePrecisionAndRecall (via pr or dumpAccuracy) on a GeneralDataset containing at least one RVFDatum constructed without a label, or whose label was set to null.","commonSituations":"Building datasets manually with new RVFDatum(features, null); reading test files missing the gold-answer column; datasets converted from unlabeled sources before evaluation.","solutions":["Ensure every datum in testData has a gold label before evaluation (fix the datum construction or input file)","Filter out unlabeled datums into a separate set and evaluate only labeled ones","Log/inspect the offending datum (its toString is in the message) to find where the label was lost"],"exampleFix":"// before\ndataset.add(new RVFDatum<>(features, null));\n// after\ndataset.add(new RVFDatum<>(features, goldLabel)); // label from the data file","handlingStrategy":"validation","validationCode":"for (RVFDatum<L,F> d : testData) { if (d.label() == null) throw new IllegalStateException(\"Unlabeled datum: \" + d); }","typeGuard":"boolean isLabeled(RVFDatum<L,F> d) { return d.label() != null; }","tryCatchPattern":"try { pr = clf.evaluatePrecisionAndRecall(test, target); } catch (IllegalArgumentException e) { log.error(\"Dataset has unlabeled datums\", e); }","preventionTips":["Validate gold labels at dataset construction/ingest time","Keep the gold-answer column in test TSV files","Filter unlabeled examples into a separate diagnostic set before evaluation"],"tags":["java","classification","unlabeled-data"],"backgroundTag":"empty-required-field","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}