{"record":{"id":"c0a2e53f55fd9657","repo":"stanfordnlp/CoreNLP","slug":"token-factory-is-null","errorCode":null,"errorMessage":": Token factory is null.","messagePattern":": Token factory is null\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/international/arabic/process/ArabicLexer.java","lineNumber":1022,"sourceCode":"   if (pos == 0) return false;\n   String thisChar = String.valueOf(text.charAt(pos));\n   if (!thisChar.equals(String.valueOf(text.charAt(pos - 1))))\n     return false;\n   if (pos < text.length() - 1 && thisChar.equals(String.valueOf(text.charAt(pos + 1))))\n     return true;\n   if (pos >= 2 && thisChar.equals(String.valueOf(text.charAt(pos - 2))))\n     return true;\n   return false;\n }\n \n   /** Make the next token.\n   *\n   *  @param txt What the token should be\n   *  @param originalText The original String that got transformed into txt\n   */\n  private Object getNext(String txt, String originalText) {\n    if (tokenFactory == null) {\n      throw new RuntimeException(this.getClass().getName() + \": Token factory is null.\");\n    }\n    if (invertible) {\n      String str = prevWordAfter.toString();\n      prevWordAfter.setLength(0);\n      CoreLabel word = (CoreLabel) tokenFactory.makeToken(txt, Math.toIntExact(yychar), yylength());\n      word.set(CoreAnnotations.OriginalTextAnnotation.class, originalText);\n      word.set(CoreAnnotations.BeforeAnnotation.class, str);\n      prevWord.set(CoreAnnotations.AfterAnnotation.class, str);\n      prevWord = word;\n      return word;\n    } else {\n      return tokenFactory.makeToken(txt, Math.toIntExact(yychar), yylength());\n    }\n  }\n  \n  private Object getNext(boolean isWord) {\n    String text = yytext();\n    String normText = normalizeToken(text, isWord);","sourceCodeStart":1004,"sourceCodeEnd":1040,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/arabic/process/ArabicLexer.java#L1004-L1040","documentation":"ArabicLexer.getNext() creates tokens via its tokenFactory; if the factory was never initialized (null), it throws this RuntimeException prefixed with the lexer class name. This is an internal invariant: a lexer built through the normal constructor always sets the factory, so null means misconfigured construction.","triggerScenarios":"A subclass or reflection-constructed ArabicLexer where tokenFactory was never assigned before tokenizing; constructing the lexer with a null factory argument.","commonSituations":"Programmatic subclassing of the generated lexer, custom TokenizerFactory implementations that pass null factories into ArabicLexer, or partially initialized lexers after a failed constructor.","solutions":["Ensure the ArabicLexer is constructed with a non-null TokenFactory (e.g. CoreLabelTokenFactory)","If subclassing, call the proper super constructor that sets tokenFactory before scanning","Use ArabicLexerFactory or ArabicTokenizer instead of instantiating the lexer directly"],"exampleFix":"// before\nArabicLexer lexer = new ArabicLexer(reader, options, false, \"UTF-8\", props, null);\n// after\nArabicLexer lexer = new ArabicLexer(reader, options, false, \"UTF-8\", props, new CoreLabelTokenFactory());","handlingStrategy":"type-guard","validationCode":"if (lexerFactory == null) {\n  throw new IllegalArgumentException(\"ArabicLexer requires a non-null TokenFactory\");\n}","typeGuard":"boolean factoryReady(TokenFactory tf) { return tf != null; }","tryCatchPattern":"try {\n  String tok = (String) lexer.next();\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().endsWith(\": Token factory is null.\")) {\n    throw new IllegalStateException(\"Lexer misconfigured: construct with a TokenFactory\", e);\n  }\n  throw e;\n}","preventionTips":["Never pass null as the factory argument to ArabicLexer","Construct lexers via ArabicLexerFactory/ArabicTokenizer","When subclassing, ensure super constructor sets the factory"],"tags":["arabic-nlp","tokenization","null-check"],"backgroundTag":"null-argument","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}