{"record":{"id":"e9604feabade8993","repo":"stanfordnlp/CoreNLP","slug":"could-not-load-tokenizer-factory","errorCode":null,"errorMessage":"Could not load tokenizer factory","messagePattern":"Could not load tokenizer factory","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/tagger/maxent/MaxentTagger.java","lineNumber":598,"sourceCode":"                                  config.getTokenizerFactory(),\n                                  config.getTokenizerOptions(),\n                                  config.getTokenizerInvertible());\n  }\n\n  protected static TokenizerFactory<? extends HasWord>\n    chooseTokenizerFactory(boolean tokenize, String tokenizerFactory,\n                           String tokenizerOptions, boolean invertible) {\n    if (tokenize && tokenizerFactory.trim().length() != 0) {\n      //return (TokenizerFactory<? extends HasWord>) Class.forName(getTokenizerFactory()).newInstance();\n      try {\n        @SuppressWarnings({\"unchecked\"})\n        Class<TokenizerFactory<? extends HasWord>> clazz = (Class<TokenizerFactory<? extends HasWord>>) Class.forName(tokenizerFactory.trim());\n        Method factoryMethod = clazz.getMethod(\"newTokenizerFactory\");\n        @SuppressWarnings({\"unchecked\"})\n        TokenizerFactory<? extends HasWord> factory = (TokenizerFactory<? extends HasWord>) factoryMethod.invoke(tokenizerOptions);\n        return factory;\n      } catch (Exception e) {\n        throw new RuntimeException(\"Could not load tokenizer factory\", e);\n      }\n    } else if (tokenize) {\n      if (invertible) {\n        if (tokenizerOptions.equals(\"\")) {\n          tokenizerOptions = \"invertible=true\";\n        } else if (!tokenizerOptions.matches(\"(^|.*,)invertible=true\")) {\n          tokenizerOptions += \",invertible=true\";\n        }\n        return PTBTokenizerFactory.newCoreLabelTokenizerFactory(tokenizerOptions);\n      } else {\n        return PTBTokenizerFactory.newWordTokenizerFactory(tokenizerOptions);\n      }\n    } else {\n      return WhitespaceTokenizer.factory();\n    }\n  }\n\n  /** Serialize the ExtractorFrames and ExtractorFramesRare to os. */","sourceCodeStart":580,"sourceCodeEnd":616,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/tagger/maxent/MaxentTagger.java#L580-L616","documentation":"MaxentTagger.chooseTokenizerFactory loads a custom tokenizer factory by reflection: Class.forName on the configured class, then invokes its static newTokenizerFactory() method. Any failure (class missing, no such method, invocation error, wrong type) is wrapped in this RuntimeException. It occurs when the model/config specifies a tokenizerFactory.","triggerScenarios":"Configuring tokenizerFactory=... with a class not on the classpath; the class lacks a public static newTokenizerFactory() method; newTokenizerFactory throws or returns a non-TokenizerFactory object; wrong tokenizerOptions cause the factory method to fail.","commonSituations":"Loading a model trained with a language-specific tokenizer (e.g. Arabic/Chinese) on a deployment without the required tokenizer classes; typos in the tokenizerFactory property; mixed CoreNLP jar versions where the factory method signature changed.","solutions":["Add the jar containing the configured tokenizer factory class to the runtime classpath (e.g. models jar for language-specific tokenizers)","Verify the class has a public static method newTokenizerFactory() (e.g. PTBTokenizerFactory, WhitespaceTokenizerFactory)","Check the tokenizerFactory property spelling and the 'caused by' exception for the exact reflection failure","If you don't need a custom tokenizer, remove the tokenizerFactory option so the default PTB/whitespace path is used"],"exampleFix":"// before (class not on classpath)\nProperties props = new Properties();\nprops.setProperty(\"tokenizerFactory\", \"edu.stanford.nlp.int Arabic.ArabicDocumentReaderAndWriter$ArabicTokenizerFactory\"); // typo/missing\n// after\nprops.setProperty(\"tokenizerFactory\", \"edu.stanford.nlp.international.Arabic.process.ArabicTokenizerFactory\");\n// and ensure stanford-corenlp models jar is on the classpath","handlingStrategy":"try-catch","validationCode":"String tf = config.getTokenize().tokenizerFactory;\nif (tf != null && !tf.isEmpty()) {\n  try {\n    Class<?> c = Class.forName(tf.trim());\n    c.getMethod(\"newTokenizerFactory\");\n  } catch (ClassNotFoundException | NoSuchMethodException e) {\n    throw new IllegalStateException(\"Tokenizer factory not loadable: \" + tf, e);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  tagger.chooseTokenizerFactory();\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().equals(\"Could not load tokenizer factory\")) {\n    // fall back to default tokenizer or fix classpath per e.getCause()\n  } else { throw e; }\n}","preventionTips":["Ship the models/language jar that provides the tokenizer factory classes at runtime","Confirm the factory class exposes public static newTokenizerFactory()","Match CoreNLP jar versions between training and deployment","If the model does not require a custom tokenizer, omit the tokenizerFactory option"],"tags":["java","reflection","tokenizer","classpath"],"backgroundTag":"class-not-found","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}