stanfordnlp/CoreNLP · error · IllegalArgumentException

No valid tokenizer type provided. Use -tokenize.language…

Error message

No valid tokenizer type provided.
Use -tokenize.language, -tokenize.class, or -tokenize.whitespace 
to specify a tokenizer.

What it means

initFactory's switch has a default branch for TokenizerType values that reached it without a valid type being determined. Only Unspecified is defaulted to PTBTokenizer; any other unrecognized type state throws IllegalArgumentException telling the user to configure tokenization explicitly.

Solutions

1. Set one of -tokenize.language, -tokenize.class, or -tokenize.whitespace=true in the properties.
2. Use the Unspecified type (or omit) to get the default PTBTokenizer.
3. Check which TokenizerType you are passing programmatically and map it to a supported one.
4. Upgrade/align CoreNLP versions so enum values match supported factory branches.

Example fix

// before
new TokenizerAnnotator(TokenizerType.HEBREW, props); // unhandled branch
// after
props.setProperty("tokenize.language", "hebrew");
new TokenizerAnnotator(props);

boolean anyTokenizerOpt = props.containsKey("tokenize.language") || props.containsKey("tokenize.class") || Boolean.parseBoolean(props.getProperty("tokenize.whitespace","false"));
if (!anyTokenizerOpt) props.setProperty("tokenize.language", "english"); // or accept PTB default

try { new TokenizerAnnotator(type, props); } catch (IllegalArgumentException e) { return new TokenizerAnnotator(TokenizerType.Unspecified, props); }

When it happens

Trigger: Constructing TokenizerAnnotator with a TokenizerType that is neither a supported language/class type nor Unspecified — typically when code passes a type enum directly via the (TokenizerType, Properties) constructor with an unusual value.

Common situations: Programmatic construction with a TokenizerType value not covered by the factory switch; library upgrades adding enum values not yet handled; building the annotator without any tokenize.* properties in an exotic code path.

Understand the failure class

Background: "X is required", "must be set", "cannot be empty": the missing-required-config error family, from Vertex AI project/location to WeChat keys — this error's family across 18 libraries.

Related errors

• TokenizerAnnotator: unknown tokenize.class property
• TokenizerAnnotator: unknown tokenize.language property
• Not a valid quotes style:
• Not a valid ellipses style:
• Not a valid ellipses style:

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/06a20e45c2dea0da. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java:328

    case Whitespace:
      boolean eolIsSignificant = Boolean.parseBoolean(props.getProperty(EOL_PROPERTY, "false"));
      eolIsSignificant = eolIsSignificant || KEEP_NL_OPTION.equals(computeExtraOptions(props));
      factory = new WhitespaceTokenizer.WhitespaceTokenizerFactory<>(new CoreLabelTokenFactory(), eolIsSignificant);
      break;

    case English:
    case German:
      factory = PTBTokenizer.factory(new CoreLabelTokenFactory(), options);
      break;

    case Unspecified:
      log.info("No tokenizer type provided. Defaulting to PTBTokenizer.");
      factory = PTBTokenizer.factory(new CoreLabelTokenFactory(), options);
      break;

    default:
      throw new IllegalArgumentException("No valid tokenizer type provided.\n" +
                                         "Use -tokenize.language, -tokenize.class, or -tokenize.whitespace \n" +
                                         "to specify a tokenizer.");
    }
    return factory;
  }

  /**
   * Returns a thread-safe tokenizer
   */
  public Tokenizer<CoreLabel> getTokenizer(Reader r) {
    return factory.getTokenizer(r);
  }

  /**
   * Helper method to set the TokenBeginAnnotation and TokenEndAnnotation of every token.
   */
  private static void setTokenBeginTokenEnd(List<CoreLabel> tokensList) {
    int tokenIndex = 0;

View on GitHub (pinned to 1b7edd19c4)