What causes "Tokenization model was not specified in ${props}" in stanfordnlp/CoreNLP?

Set the model property, e.g. props.setProperty("stat.tokenization.model", "models/fr/statFrench.tok.model")

stanfordnlp/CoreNLP · error · IllegalArgumentException

Tokenization model was not specified in

Error message

Tokenization model was not specified in ${props}

What it means

StatTokSentAnnotator requires a trained statistical tokenization model supplied via the <name>.model property. The constructor throws IllegalArgumentException when the property is missing/null because the annotator cannot function without it.

Solutions

Set the model property, e.g. props.setProperty("stat.tokenization.model", "models/fr/statFrench.tok.model")
Download the appropriate models jar/archive containing the statistical tokenizer model for your language
Verify the property prefix matches the annotator's declared name in the pipeline

Example fix

// before
props.setProperty("annotators", "tokenize,ssplit,pos"); // statTok added without model
// after
props.setProperty("stat.model", "edu/stanford/nlp/models/sutime/.../statTokSent/model.ser");
props.setProperty("annotators", "tokenize,stat,ssplit,pos");

Defensive patterns

Strategy: validation

Validate before calling

String name = "stat"; // the annotator's declared name
if (props.getProperty(name + ".model") == null) {
  throw new IllegalArgumentException("Set " + name + ".model before enabling the stat annotator");
}

Try / catch

try {
  pipeline = new StanfordCoreNLP(props);
} catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("Tokenization model was not specified")) {
    log.error("Add e.g. props.setProperty(\"stat.model\", \"<path to model>\")");
    throw e;
  } else throw e;
}

Prevention

Set the <annotator-name>.model property whenever enabling statTokSent
Download the required statistical tokenizer models for your language
Verify the property prefix matches the annotator name registered in the pipeline
Validate required model files exist on disk/classpath before building the pipeline

When it happens

Trigger: Adding 'stat(tok|sent)' style annotator to the pipeline without defining e.g. stat.tokenization.model (or the prefixed .model key) in the Properties passed to the constructor/pipeline.

Common situations: Using the SUTime/statistical tokenizer for languages like French/Spanish without downloading the required models; copy-pasting annotator lists without the accompanying properties; property prefix mismatch (annotator name differs from properties prefix).

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/f095e252d767c0c0. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/StatTokSentAnnotator.java:50

public class StatTokSentAnnotator implements Annotator{

  StatTokSent statTokSent;

  public StatTokSentAnnotator(Properties props) {
    this(Annotator.STANFORD_CDC_TOKENIZE, props);
  }

  /** The main method to intialize a tokenizer object.*/
  public StatTokSentAnnotator(String name, Properties props) {
    // Get model and rule based tokens file paths from props
    String modelFile            = props.getProperty(name + ".model", null);
    String multiWordRulesFile   = props.getProperty(name + ".multiWordRules", null);

    // If the model is not found, throws an exception.
    // If the multi-word tokens file is not found, initialize tokenizer with empty map
    if (modelFile == null) {
      throw new IllegalArgumentException("Tokenization model was not specified in "+ props);
    }

    if (multiWordRulesFile != null){
      statTokSent = new StatTokSent(modelFile, multiWordRulesFile);
    } else {
      statTokSent = new StatTokSent(modelFile);
    }
  }

  /**
   * set isNewline()
   */
  private static void setNewlineStatus(List<CoreLabel> tokensList) {
    // label newlines
    // TODO: refactor with TokenizeAnnotator
    for (CoreLabel token : tokensList) {
      if (token.word().equals(AbstractTokenizer.NEWLINE_TOKEN) && (token.endPosition() - token.beginPosition() == 1))
        token.set(CoreAnnotations.IsNewlineAnnotation.class, true);

View on GitHub (pinned to 1b7edd19c4)