stanfordnlp/CoreNLP · error · RuntimeException

Could not load tokenizer factory

Error message

Could not load tokenizer factory

What it means

MaxentTagger.chooseTokenizerFactory loads a custom tokenizer factory by reflection: Class.forName on the configured class, then invokes its static newTokenizerFactory() method. Any failure (class missing, no such method, invocation error, wrong type) is wrapped in this RuntimeException. It occurs when the model/config specifies a tokenizerFactory.

Solutions

1. Add the jar containing the configured tokenizer factory class to the runtime classpath (e.g. models jar for language-specific tokenizers)
2. Verify the class has a public static method newTokenizerFactory() (e.g. PTBTokenizerFactory, WhitespaceTokenizerFactory)
3. Check the tokenizerFactory property spelling and the 'caused by' exception for the exact reflection failure
4. If you don't need a custom tokenizer, remove the tokenizerFactory option so the default PTB/whitespace path is used

Example fix

// before (class not on classpath)
Properties props = new Properties();
props.setProperty("tokenizerFactory", "edu.stanford.nlp.int Arabic.ArabicDocumentReaderAndWriter$ArabicTokenizerFactory"); // typo/missing
// after
props.setProperty("tokenizerFactory", "edu.stanford.nlp.international.Arabic.process.ArabicTokenizerFactory");
// and ensure stanford-corenlp models jar is on the classpath

String tf = config.getTokenize().tokenizerFactory;
if (tf != null && !tf.isEmpty()) {
  try {
    Class<?> c = Class.forName(tf.trim());
    c.getMethod("newTokenizerFactory");
  } catch (ClassNotFoundException | NoSuchMethodException e) {
    throw new IllegalStateException("Tokenizer factory not loadable: " + tf, e);
  }
}

try {
  tagger.chooseTokenizerFactory();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().equals("Could not load tokenizer factory")) {
    // fall back to default tokenizer or fix classpath per e.getCause()
  } else { throw e; }
}

When it happens

Trigger: Configuring tokenizerFactory=... with a class not on the classpath; the class lacks a public static newTokenizerFactory() method; newTokenizerFactory throws or returns a non-TokenizerFactory object; wrong tokenizerOptions cause the factory method to fail.

Common situations: Loading a model trained with a language-specific tokenizer (e.g. Arabic/Chinese) on a deployment without the required tokenizer classes; typos in the tokenizerFactory property; mixed CoreNLP jar versions where the factory method signature changed.

Related errors

• Loading: failed with
• Class is in classpath multiple times
• Could not set option
• Class at path
• Could not load class in jar:

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/e9604feabade8993. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/tagger/maxent/MaxentTagger.java:598

                                  config.getTokenizerFactory(),
                                  config.getTokenizerOptions(),
                                  config.getTokenizerInvertible());
  }

  protected static TokenizerFactory<? extends HasWord>
    chooseTokenizerFactory(boolean tokenize, String tokenizerFactory,
                           String tokenizerOptions, boolean invertible) {
    if (tokenize && tokenizerFactory.trim().length() != 0) {
      //return (TokenizerFactory<? extends HasWord>) Class.forName(getTokenizerFactory()).newInstance();
      try {
        @SuppressWarnings({"unchecked"})
        Class<TokenizerFactory<? extends HasWord>> clazz = (Class<TokenizerFactory<? extends HasWord>>) Class.forName(tokenizerFactory.trim());
        Method factoryMethod = clazz.getMethod("newTokenizerFactory");
        @SuppressWarnings({"unchecked"})
        TokenizerFactory<? extends HasWord> factory = (TokenizerFactory<? extends HasWord>) factoryMethod.invoke(tokenizerOptions);
        return factory;
      } catch (Exception e) {
        throw new RuntimeException("Could not load tokenizer factory", e);
      }
    } else if (tokenize) {
      if (invertible) {
        if (tokenizerOptions.equals("")) {
          tokenizerOptions = "invertible=true";
        } else if (!tokenizerOptions.matches("(^|.*,)invertible=true")) {
          tokenizerOptions += ",invertible=true";
        }
        return PTBTokenizerFactory.newCoreLabelTokenizerFactory(tokenizerOptions);
      } else {
        return PTBTokenizerFactory.newWordTokenizerFactory(tokenizerOptions);
      }
    } else {
      return WhitespaceTokenizer.factory();
    }
  }

  /** Serialize the ExtractorFrames and ExtractorFramesRare to os. */

View on GitHub (pinned to 1b7edd19c4)