stanfordnlp/CoreNLP · error · IllegalArgumentException

TokenizerAnnotator: unknown tokenize.language property

Error message

TokenizerAnnotator: unknown tokenize.language property ${language}

What it means

Thrown by TokenizerAnnotator.getTokenizerType when the 'tokenize.language' property contains a language name that is not in nameToTokenizerMap. The lookup uppercases the value and matches against known language names mapped to TokenizerType enums. Unknown values cause an IllegalArgumentException.

Solutions

1. Use a language name supported by nameToTokenizerMap (e.g. "english", "arabic", "chinese", "french", "german", "spanish").
2. Print or inspect TokenizerAnnotator.TokenizerType to see valid values for your version.
3. If a tokenizer class is what you have, set tokenize.class instead.
4. Omit tokenize.language to get the Unspecified default (PTBTokenizer).

Example fix

// before
props.setProperty("tokenize.language", "en_US");
// after
props.setProperty("tokenize.language", "english");

String lang = props.getProperty("tokenize.language");
if (lang != null && java.util.Arrays.stream(TokenizerAnnotator.TokenizerType.values()).noneMatch(t -> t.name().equalsIgnoreCase(lang)))
  throw new IllegalArgumentException("Unsupported tokenize.language: " + lang);

boolean isSupportedLanguage(String l) { return l != null && java.util.Arrays.stream(TokenizerAnnotator.TokenizerType.values()).anyMatch(t -> t.name().equalsIgnoreCase(l)); }

try { new TokenizerAnnotator(props); } catch (IllegalArgumentException e) { throw new IllegalArgumentException("Invalid tokenize.language '" + props.getProperty("tokenize.language") + "'", e); }

When it happens

Trigger: Setting tokenize.language to a string not present in nameToTokenizerMap, e.g. "portuguese" if unsupported, misspellings like "englsih", or locale-style codes like "en_US" that are not registered.

Common situations: Users specifying ISO codes or display names not matching the supported set; configs copied between CoreNLP versions where the supported language list changed; confusion between tokenize.language and tokenize.class keys.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• TokenizerAnnotator: unknown tokenize.class property
• No valid tokenizer type provided. Use -tokenize.language…
• Not a valid quotes style:
• Not a valid ellipses style:
• Not a valid ellipses style:

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/f3d11b760cdec706. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java:115

      boolean whitespace = Boolean.parseBoolean(props.getProperty("tokenize.whitespace", "false"));
      String language = props.getProperty("tokenize.language", "en");

      if(whitespace) {
        return Whitespace;
      }

      if (tokClass != null) {
        TokenizerType type = classToTokenizerMap.get(tokClass.toUpperCase());
        if (type == null) {
          throw new IllegalArgumentException("TokenizerAnnotator: unknown tokenize.class property " + tokClass);
        }
        return type;
      }

      if (language != null) {
        TokenizerType type = nameToTokenizerMap.get(language.toUpperCase());
        if (type == null) {
          throw new IllegalArgumentException("TokenizerAnnotator: unknown tokenize.language property " + language);
        }
        return type;
      }

      return Unspecified;
    }
  } // end enum TokenizerType


  @SuppressWarnings("WeakerAccess")
  public static final String EOL_PROPERTY = "tokenize.keepeol";
  @SuppressWarnings("WeakerAccess")
  public static final String KEEP_NL_OPTION = "tokenizeNLs,";

  private final boolean VERBOSE;
  private final TokenizerFactory<CoreLabel> factory;

  /** new segmenter properties **/

View on GitHub (pinned to 1b7edd19c4)