stanfordnlp/CoreNLP · error · RuntimeException

: Token factory is null.

Error message

: Token factory is null.

What it means

ArabicLexer.getNext() creates tokens via its tokenFactory; if the factory was never initialized (null), it throws this RuntimeException prefixed with the lexer class name. This is an internal invariant: a lexer built through the normal constructor always sets the factory, so null means misconfigured construction.

Solutions

1. Ensure the ArabicLexer is constructed with a non-null TokenFactory (e.g. CoreLabelTokenFactory)
2. If subclassing, call the proper super constructor that sets tokenFactory before scanning
3. Use ArabicLexerFactory or ArabicTokenizer instead of instantiating the lexer directly

Example fix

// before
ArabicLexer lexer = new ArabicLexer(reader, options, false, "UTF-8", props, null);
// after
ArabicLexer lexer = new ArabicLexer(reader, options, false, "UTF-8", props, new CoreLabelTokenFactory());

if (lexerFactory == null) {
  throw new IllegalArgumentException("ArabicLexer requires a non-null TokenFactory");
}

boolean factoryReady(TokenFactory tf) { return tf != null; }

try {
  String tok = (String) lexer.next();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().endsWith(": Token factory is null.")) {
    throw new IllegalStateException("Lexer misconfigured: construct with a TokenFactory", e);
  }
  throw e;
}

When it happens

Trigger: A subclass or reflection-constructed ArabicLexer where tokenFactory was never assigned before tokenizing; constructing the lexer with a null factory argument.

Common situations: Programmatic subclassing of the generated lexer, custom TokenizerFactory implementations that pass null factories into ArabicLexer, or partially initialized lexers after a failed constructor.

Related errors

• ArabicLexer: the invertible option requires a…
• Error: could not match input
• Messy token:
• Invalid mapping line:
• Cannot use custom feature factory with localFeaturesOnly…

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/c0a2e53f55fd9657. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/international/arabic/process/ArabicLexer.java:1022

   if (pos == 0) return false;
   String thisChar = String.valueOf(text.charAt(pos));
   if (!thisChar.equals(String.valueOf(text.charAt(pos - 1))))
     return false;
   if (pos < text.length() - 1 && thisChar.equals(String.valueOf(text.charAt(pos + 1))))
     return true;
   if (pos >= 2 && thisChar.equals(String.valueOf(text.charAt(pos - 2))))
     return true;
   return false;
 }
 
   /** Make the next token.
   *
   *  @param txt What the token should be
   *  @param originalText The original String that got transformed into txt
   */
  private Object getNext(String txt, String originalText) {
    if (tokenFactory == null) {
      throw new RuntimeException(this.getClass().getName() + ": Token factory is null.");
    }
    if (invertible) {
      String str = prevWordAfter.toString();
      prevWordAfter.setLength(0);
      CoreLabel word = (CoreLabel) tokenFactory.makeToken(txt, Math.toIntExact(yychar), yylength());
      word.set(CoreAnnotations.OriginalTextAnnotation.class, originalText);
      word.set(CoreAnnotations.BeforeAnnotation.class, str);
      prevWord.set(CoreAnnotations.AfterAnnotation.class, str);
      prevWord = word;
      return word;
    } else {
      return tokenFactory.makeToken(txt, Math.toIntExact(yychar), yylength());
    }
  }
  
  private Object getNext(boolean isWord) {
    String text = yytext();
    String normText = normalizeToken(text, isWord);

View on GitHub (pinned to 1b7edd19c4)