stanfordnlp/CoreNLP · error · IllegalArgumentException

TokenizerAnnotator: unknown tokenize.class property

Error message

TokenizerAnnotator: unknown tokenize.class property ${tokClass}

What it means

Thrown by TokenizerAnnotator.getTokenizerType when the 'tokenize.class' property names a tokenizer class that is not registered in the static classToTokenizerMap. The map only maps a fixed set of known tokenizer class names to TokenizerType enums. Any unrecognized class string is rejected eagerly with IllegalArgumentException.

Solutions

Check the classToTokenizerMap in TokenizerAnnotator for the exact registered class names and use one verbatim (lookup is uppercased, so case does not matter).
If you meant a language, set tokenize.language instead of tokenize.class.
Remove the tokenize.class property entirely to fall through to language/unspecified handling.
If you need whitespace tokenization, use tokenize.whitespace=true instead of a class.

Example fix

// before
props.setProperty("tokenize.class", "edu.stanford.nlp.process.PTBTokenizer");
// after
props.setProperty("tokenize.class", "PTBTokenizer");

Defensive patterns

Strategy: validation

Validate before calling

Set<String> valid = TokenizerAnnotator.getTokenizerType-scale names; // check against classToTokenizerMap
String cls = props.getProperty("tokenize.class");
if (cls != null && !Set.of("PTBTokenizer","WhitespaceTokenizer").contains(cls.toUpperCase())) throw new IllegalArgumentException("Unsupported tokenize.class: " + cls);

Type guard

boolean isValidTokenizerClass(String c) { return c != null && java.util.Arrays.stream(TokenizerAnnotator.TokenizerType.values()).anyMatch(t -> t.name().equals(c.toUpperCase())); }

Try / catch

try { new TokenizerAnnotator(props); } catch (IllegalArgumentException e) { log.error("Bad tokenize.class: " + props.getProperty("tokenize.class")); throw new ConfigException(e); }

Prevention

Validate tokenize.* properties against supported values at config load time
Prefer tokenize.language over tokenize.class unless you need a specific class
Write a startup smoke test that constructs the pipeline from your props file

When it happens

Trigger: Passing an Annotation pipeline a Properties object with tokenize.class set to a string that does not case-insensitively match an entry in classToTokenizerMap (e.g. a typo or fully-qualified class name instead of the short registered name).

Common situations: Copy-pasted pipeline configs from other projects; renaming after a CoreNLP version changed supported tokenizer classes; mixing up tokenize.class with tokenize.language values.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/f82f6a51445c55dc. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java:107

    /**
     * Get TokenizerType based on what's in the properties.
     *
     * @param props Properties to find tokenizer options in
     * @return An element of the TokenizerType enum indicating the tokenizer to use
     */
    public static TokenizerType getTokenizerType(Properties props) {
      String tokClass = props.getProperty("tokenize.class", null);
      boolean whitespace = Boolean.parseBoolean(props.getProperty("tokenize.whitespace", "false"));
      String language = props.getProperty("tokenize.language", "en");

      if(whitespace) {
        return Whitespace;
      }

      if (tokClass != null) {
        TokenizerType type = classToTokenizerMap.get(tokClass.toUpperCase());
        if (type == null) {
          throw new IllegalArgumentException("TokenizerAnnotator: unknown tokenize.class property " + tokClass);
        }
        return type;
      }

      if (language != null) {
        TokenizerType type = nameToTokenizerMap.get(language.toUpperCase());
        if (type == null) {
          throw new IllegalArgumentException("TokenizerAnnotator: unknown tokenize.language property " + language);
        }
        return type;
      }

      return Unspecified;
    }
  } // end enum TokenizerType


  @SuppressWarnings("WeakerAccess")

View on GitHub (pinned to 1b7edd19c4)