What causes ": Token factory is null." in stanfordnlp/CoreNLP?

Construct the lexer with a non-null TokenFactory (typically CoreLabelTokenFactory)

stanfordnlp/CoreNLP · critical · RuntimeException

: Token factory is null.

Error message

: Token factory is null.

What it means

ArabicLexer.getNext creates tokens lazily during scanning; it requires an initialized tokenFactory. If tokenFactory is null at that point (lexer constructed without a factory), a RuntimeException naming the lexer class with ': Token factory is null.' is thrown mid-tokenization.

Solutions

Construct the lexer with a non-null TokenFactory (typically CoreLabelTokenFactory)
Use ArabicTokenizer.getTokenizer(...) which wires the factory correctly
Before tokenizing, assert the factory is set (e.g. via the lexer constructor API that accepts a factory)

Example fix

// before
ArabicLexer lexer = new ArabicLexer(reader); // no factory
// after
ArabicLexer lexer = new ArabicLexer(reader, new CoreLabelTokenFactory(), false);

Defensive patterns

Strategy: type-guard

Validate before calling

LexedTokenFactory<?> f = getFactory();
if (f == null) {
  f = new CoreLabelTokenFactory();
}
ArabicLexer lexer = new ArabicLexer(reader, f, invertible);

Type guard

boolean hasTokenFactory(ArabicLexer lexer) {
  try { lexer.yylex(); return true; }
  catch (RuntimeException e) { return !e.getMessage().contains("Token factory is null"); }
}

Try / catch

try {
  Object tok = lexer.getNext(txt, originalText);
} catch (RuntimeException e) {
  if (e.getMessage().endsWith("Token factory is null.")) {
    throw new IllegalStateException("ArabicLexer built without a TokenFactory; use ArabicTokenizer.getTokenizer", e);
  } else throw e;
}

Prevention

Never instantiate the JFlex-generated ArabicLexer directly; use ArabicTokenizer.getTokenizer
Always pass a CoreLabelTokenFactory (or compatible factory) to lexer constructors
Add an assert factory != null right after construction in wrapper code

When it happens

Trigger: Scanning Arabic text with a lexer instance whose tokenFactory was never set (constructed without a factory and then used to tokenize).

Common situations: Programmatic instantiation of the JFlex-generated lexer without supplying a LexedTokenFactory; custom pipelines constructing ArabicLexer directly instead of via ArabicTokenizer.getTokenizer.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/751bc1ef91c4b6d2. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/international/arabic/process/ArabicLexer.flex:287

   if (pos == 0) return false;
   String thisChar = String.valueOf(text.charAt(pos));
   if (!thisChar.equals(String.valueOf(text.charAt(pos - 1))))
     return false;
   if (pos < text.length() - 1 && thisChar.equals(String.valueOf(text.charAt(pos + 1))))
     return true;
   if (pos >= 2 && thisChar.equals(String.valueOf(text.charAt(pos - 2))))
     return true;
   return false;
 }
 
   /** Make the next token.
   *
   *  @param txt What the token should be
   *  @param originalText The original String that got transformed into txt
   */
  private Object getNext(String txt, String originalText) {
    if (tokenFactory == null) {
      throw new RuntimeException(this.getClass().getName() + ": Token factory is null.");
    }
    if (invertible) {
      String str = prevWordAfter.toString();
      prevWordAfter.setLength(0);
      CoreLabel word = (CoreLabel) tokenFactory.makeToken(txt, Math.toIntExact(yychar), yylength());
      word.set(CoreAnnotations.OriginalTextAnnotation.class, originalText);
      word.set(CoreAnnotations.BeforeAnnotation.class, str);
      prevWord.set(CoreAnnotations.AfterAnnotation.class, str);
      prevWord = word;
      return word;
    } else {
      return tokenFactory.makeToken(txt, Math.toIntExact(yychar), yylength());
    }
  }
  
  private Object getNext(boolean isWord) {
    String text = yytext();
    String normText = normalizeToken(text, isWord);

View on GitHub (pinned to 1b7edd19c4)