stanfordnlp/CoreNLP · error

Training of the CDC failed! Unable to build StatTokSent…

Error message

Training of the CDC failed!  Unable to build StatTokSent model

What it means

After the argument validation, StatTokSentTrainer invokes cdc.trainClassifier() to train the CDC (character-based classifier) on the IOB training file. A false return signals training failed internally (bad data, convergence, or I/O problem), and the tool logs this error and aborts before serialization. No StatTokSent model is produced.

Solutions

1. Inspect the IOB training file: correct format (token and label columns), non-empty, consistent encoding (UTF-8).
2. Check the path passed to -trainFileIOB is a readable regular file.
3. Validate the file on a small sample first to isolate format vs data-volume issues.
4. If training still fails, capture any underlying logged cause from the classifier and fix it (e.g. feature/label extraction errors).

Example fix

// before: empty/invalid IOB
head -n 0 train.iob > clean.iob
// after: supply a well-formed non-empty IOB file
java -cp ... StatTokSentTrainer -trainFileIOB properly_formed.iob -serializeTo model.gz

File f = new File(props.getProperty("trainFileIOB"));
if (!f.isFile() || f.length() == 0) throw new IllegalArgumentException("Empty/invalid IOB file: " + f);
try (BufferedReader r = Files.newBufferedReader(f.toPath(), StandardCharsets.UTF_8)) {
    String line = r.readLine();
    if (line == null || line.split("\t").length < 2) throw new IllegalArgumentException("Malformed IOB columns");
}

When it happens

Trigger: Calling main with -loadClassifier unset so training runs, and cdc.trainClassifier(trainFileIOB.getPath()) returns false — e.g. malformed/empty IOB file, unreadable path, or the underlying classifier failing to train.

Common situations: Hand-converted IOB files with wrong column counts or empty class labels; a trainFileIOB pointing to a directory or empty file; encoding issues in the training corpus.

Related errors

• Attempting to remove features based on weight from a…
• Could not read from double initial weight file
• Could not read from float initial weight file
• Couldn't load classifier!
• Couldn't load classifier from

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/c1ffe34dcb7fe058. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/stattok/StatTokSentTrainer.java:502

    String testFile 			= properties.getProperty("testFile", null);
    String serializeTo 			= properties.getProperty("serializeTo", null);
    String crossValidationFoldsStr 	= properties.getProperty("crossValidationFolds", null);
    int crossValidationFolds 		= 0;
    if (crossValidationFoldsStr != null){
      crossValidationFolds = Integer.parseInt(crossValidationFoldsStr);
    }
    String loadClassifier		= properties.getProperty("loadClassifier", null); 

    if ((testFile == null && serializeTo == null && crossValidationFolds < 2) ||
        (trainFileIOB == null && loadClassifier == null)) { 
      logger.err("Not enough information provided via command line properties or properties file.  If you want to save a model, please specify -serializeTo.  Use -help for other options.");
      return;
    }

    if (loadClassifier == null) {
      if (!cdc.trainClassifier(trainFileIOB.getPath())) {
        logger.err("Training of the CDC failed!  Unable to build StatTokSent model");
        return;
      }
      serialize(serializeTo, cdc, windowSize);
    }

    if (testFile != null) {
      cdc.testClassifier(testFile);
    }
  }
}

View on GitHub (pinned to 1b7edd19c4)