{"record":{"id":"f82f6a51445c55dc","repo":"stanfordnlp/CoreNLP","slug":"tokenizerannotator-unknown-tokenize-class-propert","errorCode":null,"errorMessage":"TokenizerAnnotator: unknown tokenize.class property ${tokClass}","messagePattern":"TokenizerAnnotator: unknown tokenize\\.class property (.+?)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java","lineNumber":107,"sourceCode":"    /**\n     * Get TokenizerType based on what's in the properties.\n     *\n     * @param props Properties to find tokenizer options in\n     * @return An element of the TokenizerType enum indicating the tokenizer to use\n     */\n    public static TokenizerType getTokenizerType(Properties props) {\n      String tokClass = props.getProperty(\"tokenize.class\", null);\n      boolean whitespace = Boolean.parseBoolean(props.getProperty(\"tokenize.whitespace\", \"false\"));\n      String language = props.getProperty(\"tokenize.language\", \"en\");\n\n      if(whitespace) {\n        return Whitespace;\n      }\n\n      if (tokClass != null) {\n        TokenizerType type = classToTokenizerMap.get(tokClass.toUpperCase());\n        if (type == null) {\n          throw new IllegalArgumentException(\"TokenizerAnnotator: unknown tokenize.class property \" + tokClass);\n        }\n        return type;\n      }\n\n      if (language != null) {\n        TokenizerType type = nameToTokenizerMap.get(language.toUpperCase());\n        if (type == null) {\n          throw new IllegalArgumentException(\"TokenizerAnnotator: unknown tokenize.language property \" + language);\n        }\n        return type;\n      }\n\n      return Unspecified;\n    }\n  } // end enum TokenizerType\n\n\n  @SuppressWarnings(\"WeakerAccess\")","sourceCodeStart":89,"sourceCodeEnd":125,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java#L89-L125","documentation":"Thrown by TokenizerAnnotator.getTokenizerType when the 'tokenize.class' property names a tokenizer class that is not registered in the static classToTokenizerMap. The map only maps a fixed set of known tokenizer class names to TokenizerType enums. Any unrecognized class string is rejected eagerly with IllegalArgumentException.","triggerScenarios":"Passing an Annotation pipeline a Properties object with tokenize.class set to a string that does not case-insensitively match an entry in classToTokenizerMap (e.g. a typo or fully-qualified class name instead of the short registered name).","commonSituations":"Copy-pasted pipeline configs from other projects; renaming after a CoreNLP version changed supported tokenizer classes; mixing up tokenize.class with tokenize.language values.","solutions":["Check the classToTokenizerMap in TokenizerAnnotator for the exact registered class names and use one verbatim (lookup is uppercased, so case does not matter).","If you meant a language, set tokenize.language instead of tokenize.class.","Remove the tokenize.class property entirely to fall through to language/unspecified handling.","If you need whitespace tokenization, use tokenize.whitespace=true instead of a class."],"exampleFix":"// before\nprops.setProperty(\"tokenize.class\", \"edu.stanford.nlp.process.PTBTokenizer\");\n// after\nprops.setProperty(\"tokenize.class\", \"PTBTokenizer\");","handlingStrategy":"validation","validationCode":"Set<String> valid = TokenizerAnnotator.getTokenizerType-scale names; // check against classToTokenizerMap\nString cls = props.getProperty(\"tokenize.class\");\nif (cls != null && !Set.of(\"PTBTokenizer\",\"WhitespaceTokenizer\").contains(cls.toUpperCase())) throw new IllegalArgumentException(\"Unsupported tokenize.class: \" + cls);","typeGuard":"boolean isValidTokenizerClass(String c) { return c != null && java.util.Arrays.stream(TokenizerAnnotator.TokenizerType.values()).anyMatch(t -> t.name().equals(c.toUpperCase())); }","tryCatchPattern":"try { new TokenizerAnnotator(props); } catch (IllegalArgumentException e) { log.error(\"Bad tokenize.class: \" + props.getProperty(\"tokenize.class\")); throw new ConfigException(e); }","preventionTips":["Validate tokenize.* properties against supported values at config load time","Prefer tokenize.language over tokenize.class unless you need a specific class","Write a startup smoke test that constructs the pipeline from your props file"],"tags":["java","tokenizer","configuration","corenlp"],"backgroundTag":"invalid-config-value","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}