{"record":{"id":"751bc1ef91c4b6d2","repo":"stanfordnlp/CoreNLP","slug":"token-factory-is-null-751bc1","errorCode":null,"errorMessage":": Token factory is null.","messagePattern":": Token factory is null\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"src/edu/stanford/nlp/international/arabic/process/ArabicLexer.flex","lineNumber":287,"sourceCode":"   if (pos == 0) return false;\n   String thisChar = String.valueOf(text.charAt(pos));\n   if (!thisChar.equals(String.valueOf(text.charAt(pos - 1))))\n     return false;\n   if (pos < text.length() - 1 && thisChar.equals(String.valueOf(text.charAt(pos + 1))))\n     return true;\n   if (pos >= 2 && thisChar.equals(String.valueOf(text.charAt(pos - 2))))\n     return true;\n   return false;\n }\n \n   /** Make the next token.\n   *\n   *  @param txt What the token should be\n   *  @param originalText The original String that got transformed into txt\n   */\n  private Object getNext(String txt, String originalText) {\n    if (tokenFactory == null) {\n      throw new RuntimeException(this.getClass().getName() + \": Token factory is null.\");\n    }\n    if (invertible) {\n      String str = prevWordAfter.toString();\n      prevWordAfter.setLength(0);\n      CoreLabel word = (CoreLabel) tokenFactory.makeToken(txt, Math.toIntExact(yychar), yylength());\n      word.set(CoreAnnotations.OriginalTextAnnotation.class, originalText);\n      word.set(CoreAnnotations.BeforeAnnotation.class, str);\n      prevWord.set(CoreAnnotations.AfterAnnotation.class, str);\n      prevWord = word;\n      return word;\n    } else {\n      return tokenFactory.makeToken(txt, Math.toIntExact(yychar), yylength());\n    }\n  }\n  \n  private Object getNext(boolean isWord) {\n    String text = yytext();\n    String normText = normalizeToken(text, isWord);","sourceCodeStart":269,"sourceCodeEnd":305,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/arabic/process/ArabicLexer.flex#L269-L305","documentation":"ArabicLexer.getNext creates tokens lazily during scanning; it requires an initialized tokenFactory. If tokenFactory is null at that point (lexer constructed without a factory), a RuntimeException naming the lexer class with ': Token factory is null.' is thrown mid-tokenization.","triggerScenarios":"Scanning Arabic text with a lexer instance whose tokenFactory was never set (constructed without a factory and then used to tokenize).","commonSituations":"Programmatic instantiation of the JFlex-generated lexer without supplying a LexedTokenFactory; custom pipelines constructing ArabicLexer directly instead of via ArabicTokenizer.getTokenizer.","solutions":["Construct the lexer with a non-null TokenFactory (typically CoreLabelTokenFactory)","Use ArabicTokenizer.getTokenizer(...) which wires the factory correctly","Before tokenizing, assert the factory is set (e.g. via the lexer constructor API that accepts a factory)"],"exampleFix":"// before\nArabicLexer lexer = new ArabicLexer(reader); // no factory\n// after\nArabicLexer lexer = new ArabicLexer(reader, new CoreLabelTokenFactory(), false);","handlingStrategy":"type-guard","validationCode":"LexedTokenFactory<?> f = getFactory();\nif (f == null) {\n  f = new CoreLabelTokenFactory();\n}\nArabicLexer lexer = new ArabicLexer(reader, f, invertible);","typeGuard":"boolean hasTokenFactory(ArabicLexer lexer) {\n  try { lexer.yylex(); return true; }\n  catch (RuntimeException e) { return !e.getMessage().contains(\"Token factory is null\"); }\n}","tryCatchPattern":"try {\n  Object tok = lexer.getNext(txt, originalText);\n} catch (RuntimeException e) {\n  if (e.getMessage().endsWith(\"Token factory is null.\")) {\n    throw new IllegalStateException(\"ArabicLexer built without a TokenFactory; use ArabicTokenizer.getTokenizer\", e);\n  } else throw e;\n}","preventionTips":["Never instantiate the JFlex-generated ArabicLexer directly; use ArabicTokenizer.getTokenizer","Always pass a CoreLabelTokenFactory (or compatible factory) to lexer constructors","Add an assert factory != null right after construction in wrapper code"],"tags":["java","tokenization","arabic","null"],"backgroundTag":"null-argument","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}