{"record":{"id":"f3d11b760cdec706","repo":"stanfordnlp/CoreNLP","slug":"tokenizerannotator-unknown-tokenize-language-prop","errorCode":null,"errorMessage":"TokenizerAnnotator: unknown tokenize.language property ${language}","messagePattern":"TokenizerAnnotator: unknown tokenize\\.language property (.+?)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java","lineNumber":115,"sourceCode":"      boolean whitespace = Boolean.parseBoolean(props.getProperty(\"tokenize.whitespace\", \"false\"));\n      String language = props.getProperty(\"tokenize.language\", \"en\");\n\n      if(whitespace) {\n        return Whitespace;\n      }\n\n      if (tokClass != null) {\n        TokenizerType type = classToTokenizerMap.get(tokClass.toUpperCase());\n        if (type == null) {\n          throw new IllegalArgumentException(\"TokenizerAnnotator: unknown tokenize.class property \" + tokClass);\n        }\n        return type;\n      }\n\n      if (language != null) {\n        TokenizerType type = nameToTokenizerMap.get(language.toUpperCase());\n        if (type == null) {\n          throw new IllegalArgumentException(\"TokenizerAnnotator: unknown tokenize.language property \" + language);\n        }\n        return type;\n      }\n\n      return Unspecified;\n    }\n  } // end enum TokenizerType\n\n\n  @SuppressWarnings(\"WeakerAccess\")\n  public static final String EOL_PROPERTY = \"tokenize.keepeol\";\n  @SuppressWarnings(\"WeakerAccess\")\n  public static final String KEEP_NL_OPTION = \"tokenizeNLs,\";\n\n  private final boolean VERBOSE;\n  private final TokenizerFactory<CoreLabel> factory;\n\n  /** new segmenter properties **/","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java#L97-L133","documentation":"Thrown by TokenizerAnnotator.getTokenizerType when the 'tokenize.language' property contains a language name that is not in nameToTokenizerMap. The lookup uppercases the value and matches against known language names mapped to TokenizerType enums. Unknown values cause an IllegalArgumentException.","triggerScenarios":"Setting tokenize.language to a string not present in nameToTokenizerMap, e.g. \"portuguese\" if unsupported, misspellings like \"englsih\", or locale-style codes like \"en_US\" that are not registered.","commonSituations":"Users specifying ISO codes or display names not matching the supported set; configs copied between CoreNLP versions where the supported language list changed; confusion between tokenize.language and tokenize.class keys.","solutions":["Use a language name supported by nameToTokenizerMap (e.g. \"english\", \"arabic\", \"chinese\", \"french\", \"german\", \"spanish\").","Print or inspect TokenizerAnnotator.TokenizerType to see valid values for your version.","If a tokenizer class is what you have, set tokenize.class instead.","Omit tokenize.language to get the Unspecified default (PTBTokenizer)."],"exampleFix":"// before\nprops.setProperty(\"tokenize.language\", \"en_US\");\n// after\nprops.setProperty(\"tokenize.language\", \"english\");","handlingStrategy":"validation","validationCode":"String lang = props.getProperty(\"tokenize.language\");\nif (lang != null && java.util.Arrays.stream(TokenizerAnnotator.TokenizerType.values()).noneMatch(t -> t.name().equalsIgnoreCase(lang)))\n  throw new IllegalArgumentException(\"Unsupported tokenize.language: \" + lang);","typeGuard":"boolean isSupportedLanguage(String l) { return l != null && java.util.Arrays.stream(TokenizerAnnotator.TokenizerType.values()).anyMatch(t -> t.name().equalsIgnoreCase(l)); }","tryCatchPattern":"try { new TokenizerAnnotator(props); } catch (IllegalArgumentException e) { throw new IllegalArgumentException(\"Invalid tokenize.language '\" + props.getProperty(\"tokenize.language\") + \"'\", e); }","preventionTips":["Keep a whitelist of languages your app supports and validate user input against it","Test pipeline construction at startup, not at first request","Pin your CoreNLP version and re-validate configs on upgrades"],"tags":["java","tokenizer","configuration","corenlp"],"backgroundTag":"invalid-enum-value","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}