{"record":{"id":"18b93a241cfb9747","repo":"stanfordnlp/CoreNLP","slug":"no-multi-word-rules-provided-no-infermultiwordrul","errorCode":null,"errorMessage":"No multi-word rules provided. No inferMultiWordRules flag validated. Not inferring rules from training.","messagePattern":"No multi-word rules provided\\. No inferMultiWordRules flag validated\\. Not inferring rules from training\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/edu/stanford/nlp/process/stattok/StatTokSentTrainer.java","lineNumber":439,"sourceCode":"      logger.err(\"Error: No training file provided in properties or via command line.\");\n      return;\n    }\n\n    Map<String, String[]> multiWordRules = new HashMap<String, String[]>();\n    // Read or generate multi-word rules\n    if (multiWordRulesFile != null){\n      if (inferMultiWordRules){\n        logger.warn(\"Conflicting properties. Multi-word rules file will be considered.\");\n      }\n      logger.info(\"Reading Multi-Word rules file ... \");\n      multiWordRules = trainTokenizer.readMultiWordRules(multiWordRulesFile);\n    } else {\n      if (inferMultiWordRules){\n        logger.info(\"Inferring Multi-Word rules from training set ... \");\n        multiWordRules = trainTokenizer.inferMultiWordRules(trainFile);\n      }\n      else{\n        logger.warn(\"No multi-word rules provided. No inferMultiWordRules flag validated. Not inferring rules from training.\");\n      }\n    }\n\n    // Generate training file from ConLL-U\n    logger.info(\"Creating training set from \"+trainFile);\n    ArrayList<Pair<String, String>> classCharText = trainTokenizer.fileToTrainSet(trainFile,multiWordRules);\n    logger.info(\"Adding Features\");\n    List<String> trainingInput = trainTokenizer.addFeatures(classCharText, windowSize);\n    logger.info(\"Training is ready.\");\n\t\t\n    // Write temporary training data on file\n    File trainFileIOB = File.createTempFile(\"training.\", \".iob\");\n    trainFileIOB.deleteOnExit();\n    OutputStreamWriter fileWriter = new OutputStreamWriter(new FileOutputStream(trainFileIOB), StandardCharsets.UTF_8);\n    for (String line : trainingInput){\n      fileWriter.write(line+System.lineSeparator());\n    }\n    fileWriter.close();","sourceCodeStart":421,"sourceCodeEnd":457,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/process/stattok/StatTokSentTrainer.java#L421-L457","documentation":"When StatTokSentTrainer has no multiWordRulesFile and the inferMultiWordRules flag is not enabled, it cannot obtain multi-word tokenization rules and logs this warning, continuing with an empty rule map. Training proceeds but multi-word expressions (e.g. 'del', 'au') will not be handled, degrading tokenizer quality on languages that need them.","triggerScenarios":"Running StatTokSentTrainer with neither -multiWordRulesFile nor the inferMultiWordRules option set true, hitting the else-branch warn at line 439.","commonSituations":"Training French/Spanish/Galician tokenizers where multi-word rules matter but the flag was forgotten; users who read BuildMultiWordRules output but forgot to pass the resulting file; misreading the flag name and passing an unrecognized variant.","solutions":["Pass -multiWordRulesFile pointing to rules generated by BuildMultiWordRules.","Or enable the inferMultiWordRules flag so rules are inferred from the training set automatically.","Generate rules first with BuildMultiWordRules if you don't have a rules file.","Accept the warning only if the target language truly has no multi-word tokens."],"exampleFix":"// before\njava -cp ... StatTokSentTrainer -trainFile train.conllu -serializeTo model.gz\n// after\njava -cp ... StatTokSentTrainer -trainFile train.conllu -multiWordRulesFile mw-rules.txt -serializeTo model.gz","handlingStrategy":"validation","validationCode":"boolean hasRules = argsContains(args, \"multiWordRulesFile\");\nboolean infer = Boolean.parseBoolean(props.getProperty(\"inferMultiWordRules\", \"false\"));\nif (!hasRules && !infer) {\n    System.err.println(\"WARN: no multi-word rules source; pass -multiWordRulesFile or enable inferMultiWordRules\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["For languages with contractions (fr, es, pt, gl), always supply multi-word rules.","Generate rules with BuildMultiWordRules as a pipeline step before training.","Treat this warning as fatal in CI for rule-sensitive languages."],"tags":["java","training","tokenization","configuration"],"backgroundTag":"missing-required-flag","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}