{"record":{"id":"06a20e45c2dea0da","repo":"stanfordnlp/CoreNLP","slug":"no-valid-tokenizer-type-provided-use-tokenize-la","errorCode":null,"errorMessage":"No valid tokenizer type provided.\nUse -tokenize.language, -tokenize.class, or -tokenize.whitespace \nto specify a tokenizer.","messagePattern":"No valid tokenizer type provided\\.\nUse -tokenize\\.language, -tokenize\\.class, or -tokenize\\.whitespace \nto specify a tokenizer\\.","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java","lineNumber":328,"sourceCode":"\n    case Whitespace:\n      boolean eolIsSignificant = Boolean.parseBoolean(props.getProperty(EOL_PROPERTY, \"false\"));\n      eolIsSignificant = eolIsSignificant || KEEP_NL_OPTION.equals(computeExtraOptions(props));\n      factory = new WhitespaceTokenizer.WhitespaceTokenizerFactory<>(new CoreLabelTokenFactory(), eolIsSignificant);\n      break;\n\n    case English:\n    case German:\n      factory = PTBTokenizer.factory(new CoreLabelTokenFactory(), options);\n      break;\n\n    case Unspecified:\n      log.info(\"No tokenizer type provided. Defaulting to PTBTokenizer.\");\n      factory = PTBTokenizer.factory(new CoreLabelTokenFactory(), options);\n      break;\n\n    default:\n      throw new IllegalArgumentException(\"No valid tokenizer type provided.\\n\" +\n                                         \"Use -tokenize.language, -tokenize.class, or -tokenize.whitespace \\n\" +\n                                         \"to specify a tokenizer.\");\n    }\n    return factory;\n  }\n\n  /**\n   * Returns a thread-safe tokenizer\n   */\n  public Tokenizer<CoreLabel> getTokenizer(Reader r) {\n    return factory.getTokenizer(r);\n  }\n\n  /**\n   * Helper method to set the TokenBeginAnnotation and TokenEndAnnotation of every token.\n   */\n  private static void setTokenBeginTokenEnd(List<CoreLabel> tokensList) {\n    int tokenIndex = 0;","sourceCodeStart":310,"sourceCodeEnd":346,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java#L310-L346","documentation":"initFactory's switch has a default branch for TokenizerType values that reached it without a valid type being determined. Only Unspecified is defaulted to PTBTokenizer; any other unrecognized type state throws IllegalArgumentException telling the user to configure tokenization explicitly.","triggerScenarios":"Constructing TokenizerAnnotator with a TokenizerType that is neither a supported language/class type nor Unspecified — typically when code passes a type enum directly via the (TokenizerType, Properties) constructor with an unusual value.","commonSituations":"Programmatic construction with a TokenizerType value not covered by the factory switch; library upgrades adding enum values not yet handled; building the annotator without any tokenize.* properties in an exotic code path.","solutions":["Set one of -tokenize.language, -tokenize.class, or -tokenize.whitespace=true in the properties.","Use the Unspecified type (or omit) to get the default PTBTokenizer.","Check which TokenizerType you are passing programmatically and map it to a supported one.","Upgrade/align CoreNLP versions so enum values match supported factory branches."],"exampleFix":"// before\nnew TokenizerAnnotator(TokenizerType.HEBREW, props); // unhandled branch\n// after\nprops.setProperty(\"tokenize.language\", \"hebrew\");\nnew TokenizerAnnotator(props);","handlingStrategy":"validation","validationCode":"boolean anyTokenizerOpt = props.containsKey(\"tokenize.language\") || props.containsKey(\"tokenize.class\") || Boolean.parseBoolean(props.getProperty(\"tokenize.whitespace\",\"false\"));\nif (!anyTokenizerOpt) props.setProperty(\"tokenize.language\", \"english\"); // or accept PTB default","typeGuard":null,"tryCatchPattern":"try { new TokenizerAnnotator(type, props); } catch (IllegalArgumentException e) { return new TokenizerAnnotator(TokenizerType.Unspecified, props); }","preventionTips":["Always set an explicit tokenizer option in pipeline configs","Never construct TokenizerAnnotator with raw enum values not originating from getTokenizerType","Cover annotator construction in unit tests"],"tags":["java","tokenizer","configuration","corenlp"],"backgroundTag":"missing-required-config","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}