{"record":{"id":"c1ffe34dcb7fe058","repo":"stanfordnlp/CoreNLP","slug":"training-of-the-cdc-failed-unable-to-build-statt","errorCode":null,"errorMessage":"Training of the CDC failed!  Unable to build StatTokSent model","messagePattern":"Training of the CDC failed!  Unable to build StatTokSent model","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/process/stattok/StatTokSentTrainer.java","lineNumber":502,"sourceCode":"\n    String testFile \t\t\t= properties.getProperty(\"testFile\", null);\n    String serializeTo \t\t\t= properties.getProperty(\"serializeTo\", null);\n    String crossValidationFoldsStr \t= properties.getProperty(\"crossValidationFolds\", null);\n    int crossValidationFolds \t\t= 0;\n    if (crossValidationFoldsStr != null){\n      crossValidationFolds = Integer.parseInt(crossValidationFoldsStr);\n    }\n    String loadClassifier\t\t= properties.getProperty(\"loadClassifier\", null); \n\n    if ((testFile == null && serializeTo == null && crossValidationFolds < 2) ||\n        (trainFileIOB == null && loadClassifier == null)) { \n      logger.err(\"Not enough information provided via command line properties or properties file.  If you want to save a model, please specify -serializeTo.  Use -help for other options.\");\n      return;\n    }\n\n    if (loadClassifier == null) {\n      if (!cdc.trainClassifier(trainFileIOB.getPath())) {\n        logger.err(\"Training of the CDC failed!  Unable to build StatTokSent model\");\n        return;\n      }\n      serialize(serializeTo, cdc, windowSize);\n    }\n\n    if (testFile != null) {\n      cdc.testClassifier(testFile);\n    }\n  }\n}\n","sourceCodeStart":484,"sourceCodeEnd":513,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/process/stattok/StatTokSentTrainer.java#L484-L513","documentation":"After the argument validation, StatTokSentTrainer invokes cdc.trainClassifier() to train the CDC (character-based classifier) on the IOB training file. A false return signals training failed internally (bad data, convergence, or I/O problem), and the tool logs this error and aborts before serialization. No StatTokSent model is produced.","triggerScenarios":"Calling main with -loadClassifier unset so training runs, and cdc.trainClassifier(trainFileIOB.getPath()) returns false — e.g. malformed/empty IOB file, unreadable path, or the underlying classifier failing to train.","commonSituations":"Hand-converted IOB files with wrong column counts or empty class labels; a trainFileIOB pointing to a directory or empty file; encoding issues in the training corpus.","solutions":["Inspect the IOB training file: correct format (token and label columns), non-empty, consistent encoding (UTF-8).","Check the path passed to -trainFileIOB is a readable regular file.","Validate the file on a small sample first to isolate format vs data-volume issues.","If training still fails, capture any underlying logged cause from the classifier and fix it (e.g. feature/label extraction errors)."],"exampleFix":"// before: empty/invalid IOB\nhead -n 0 train.iob > clean.iob\n// after: supply a well-formed non-empty IOB file\njava -cp ... StatTokSentTrainer -trainFileIOB properly_formed.iob -serializeTo model.gz","handlingStrategy":"validation","validationCode":"File f = new File(props.getProperty(\"trainFileIOB\"));\nif (!f.isFile() || f.length() == 0) throw new IllegalArgumentException(\"Empty/invalid IOB file: \" + f);\ntry (BufferedReader r = Files.newBufferedReader(f.toPath(), StandardCharsets.UTF_8)) {\n    String line = r.readLine();\n    if (line == null || line.split(\"\\t\").length < 2) throw new IllegalArgumentException(\"Malformed IOB columns\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate IOB format (columns, labels) before long training runs.","Train on a 100-line sample first to fail fast.","Keep training corpora in a fixed UTF-8 encoding."],"tags":["java","training","classifier","data-format"],"backgroundTag":"training-data-invalid","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}