{"record":{"id":"f095e252d767c0c0","repo":"stanfordnlp/CoreNLP","slug":"tokenization-model-was-not-specified-in-props","errorCode":null,"errorMessage":"Tokenization model was not specified in ${props}","messagePattern":"Tokenization model was not specified in (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/StatTokSentAnnotator.java","lineNumber":50,"sourceCode":"\npublic class StatTokSentAnnotator implements Annotator{\n\n  StatTokSent statTokSent;\n\n  public StatTokSentAnnotator(Properties props) {\n    this(Annotator.STANFORD_CDC_TOKENIZE, props);\n  }\n\n  /** The main method to intialize a tokenizer object.*/\n  public StatTokSentAnnotator(String name, Properties props) {\n    // Get model and rule based tokens file paths from props\n    String modelFile            = props.getProperty(name + \".model\", null);\n    String multiWordRulesFile   = props.getProperty(name + \".multiWordRules\", null);\n\n    // If the model is not found, throws an exception.\n    // If the multi-word tokens file is not found, initialize tokenizer with empty map\n    if (modelFile == null) {\n      throw new IllegalArgumentException(\"Tokenization model was not specified in \"+ props);\n    }\n\n    if (multiWordRulesFile != null){\n      statTokSent = new StatTokSent(modelFile, multiWordRulesFile);\n    } else {\n      statTokSent = new StatTokSent(modelFile);\n    }\n  }\n\n  /**\n   * set isNewline()\n   */\n  private static void setNewlineStatus(List<CoreLabel> tokensList) {\n    // label newlines\n    // TODO: refactor with TokenizeAnnotator\n    for (CoreLabel token : tokensList) {\n      if (token.word().equals(AbstractTokenizer.NEWLINE_TOKEN) && (token.endPosition() - token.beginPosition() == 1))\n        token.set(CoreAnnotations.IsNewlineAnnotation.class, true);","sourceCodeStart":32,"sourceCodeEnd":68,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/StatTokSentAnnotator.java#L32-L68","documentation":"StatTokSentAnnotator requires a trained statistical tokenization model supplied via the <name>.model property. The constructor throws IllegalArgumentException when the property is missing/null because the annotator cannot function without it.","triggerScenarios":"Adding 'stat(tok|sent)' style annotator to the pipeline without defining e.g. stat.tokenization.model (or the prefixed .model key) in the Properties passed to the constructor/pipeline.","commonSituations":"Using the SUTime/statistical tokenizer for languages like French/Spanish without downloading the required models; copy-pasting annotator lists without the accompanying properties; property prefix mismatch (annotator name differs from properties prefix).","solutions":["Set the model property, e.g. props.setProperty(\"stat.tokenization.model\", \"models/fr/statFrench.tok.model\")","Download the appropriate models jar/archive containing the statistical tokenizer model for your language","Verify the property prefix matches the annotator's declared name in the pipeline"],"exampleFix":"// before\nprops.setProperty(\"annotators\", \"tokenize,ssplit,pos\"); // statTok added without model\n// after\nprops.setProperty(\"stat.model\", \"edu/stanford/nlp/models/sutime/.../statTokSent/model.ser\");\nprops.setProperty(\"annotators\", \"tokenize,stat,ssplit,pos\");","handlingStrategy":"validation","validationCode":"String name = \"stat\"; // the annotator's declared name\nif (props.getProperty(name + \".model\") == null) {\n  throw new IllegalArgumentException(\"Set \" + name + \".model before enabling the stat annotator\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  pipeline = new StanfordCoreNLP(props);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().startsWith(\"Tokenization model was not specified\")) {\n    log.error(\"Add e.g. props.setProperty(\\\"stat.model\\\", \\\"<path to model>\\\")\");\n    throw e;\n  } else throw e;\n}","preventionTips":["Set the <annotator-name>.model property whenever enabling statTokSent","Download the required statistical tokenizer models for your language","Verify the property prefix matches the annotator name registered in the pipeline","Validate required model files exist on disk/classpath before building the pipeline"],"tags":["configuration","missing-model","tokenizer","annotator"],"backgroundTag":"missing-required-config-field","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}