stanfordnlp/CoreNLP · error · IllegalArgumentException

Cannot compute precision and recall on unlabelled dataset…

Error message

Cannot compute precision and recall on unlabelled dataset. Offending datum: ${datum}

What it means

While iterating the test dataset to compute precision/recall against a target label, the library requires every datum to carry a gold label. If datum.label() returns null for any RVFDatum, it throws an IllegalArgumentException identifying the offending datum, because unlabeled examples cannot contribute to precision/recall counts.

Solutions

1. Ensure every datum in testData has a gold label before evaluation (fix the datum construction or input file)
2. Filter out unlabeled datums into a separate set and evaluate only labeled ones
3. Log/inspect the offending datum (its toString is in the message) to find where the label was lost

Example fix

// before
dataset.add(new RVFDatum<>(features, null));
// after
dataset.add(new RVFDatum<>(features, goldLabel)); // label from the data file

for (RVFDatum<L,F> d : testData) { if (d.label() == null) throw new IllegalStateException("Unlabeled datum: " + d); }

boolean isLabeled(RVFDatum<L,F> d) { return d.label() != null; }

try { pr = clf.evaluatePrecisionAndRecall(test, target); } catch (IllegalArgumentException e) { log.error("Dataset has unlabeled datums", e); }

When it happens

Trigger: Calling evaluatePrecisionAndRecall (via pr or dumpAccuracy) on a GeneralDataset containing at least one RVFDatum constructed without a label, or whose label was set to null.

Common situations: Building datasets manually with new RVFDatum(features, null); reading test files missing the gold-answer column; datasets converted from unlabeled sources before evaluation.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• Must supply a target label to compute precision and recall…
• Line format error at line
• Error: Line has too few tab-separated columns
• Dataset could not be loaded
• Not enough columns for format

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/c77a4857357eb57c. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/classify/Classifier.java:51

   *
   * @param testData The dataset to evaluate the classifier on.
   * @param targetLabel The target label (e.g., for relation extraction, this is the relation we're interested in).
   * @return A pair of the precision (first) and recall (second) of the classifier on the target label.
   */
  public default Pair<Double, Double> evaluatePrecisionAndRecall(GeneralDataset<L, F> testData, L targetLabel) {
    if (targetLabel == null) {
      throw new IllegalArgumentException("Must supply a target label to compute precision and recall against");
    }
    // Variables to count
    int numCorrectAndTarget = 0;
    int numTargetGuess = 0;
    int numTargetGold = 0;
    // Iterate over dataset
    for (RVFDatum<L, F> datum : testData) {
      // Get the gold label
      L label = datum.label();
      if (label == null) {
        throw new IllegalArgumentException("Cannot compute precision and recall on unlabelled dataset. Offending datum: " + datum);
      }
      // Get the guess label
      L guess = classOf(datum);
      // Compute statistics on datum
      if (label.equals(targetLabel)) {
        numTargetGold += 1;
      }
      if (guess.equals(targetLabel)) {
        numTargetGuess += 1;
        if (guess.equals(label)) {
          numCorrectAndTarget += 1;
        }
      }
    }
    // Aggregate statistics
    double precision = numTargetGuess == 0 ? 0.0 : ((double) numCorrectAndTarget) / ((double) numTargetGuess);
    double recall = numTargetGold == 0 ? 1.0 : ((double) numCorrectAndTarget) / ((double) numTargetGold);
    return Pair.makePair(precision, recall);

View on GitHub (pinned to 1b7edd19c4)