stanfordnlp/CoreNLP · error · IllegalArgumentException

You can't make a Tokenizer out of a null Lexer!

Error message

You can't make a Tokenizer out of a null Lexer!

What it means

LexerTokenizer wraps a JFlex-generated Lexer; constructing it with a null Lexer would make every token request fail, so the constructor throws this IllegalArgumentException. It is a simple fail-fast guard on the mandatory lexer argument.

Solutions

1. Check why the Lexer was null: trace the factory/lookup that should have created it
2. Construct the lexer explicitly, e.g. new LexerTokenizer(new PTBLexer(new InputStreamReader(in)))
3. Verify the language/tokenizer option string maps to an available lexer class on the classpath
4. Add an assertion/null-check at the call site to fail with a more descriptive message

Example fix

// before
Lexer lex = lexerFactory.get(lang); // may be null
LexerTokenizer tok = new LexerTokenizer(lex);
// after
Lexer lex = lexerFactory.get(lang);
if (lex == null) {
  throw new IllegalStateException("No lexer registered for language: " + lang);
}
LexerTokenizer tok = new LexerTokenizer(lex);

Objects.requireNonNull(lexer, "Lexer must be created before building LexerTokenizer");

static Lexer requireLexer(Lexer l) {
  if (l == null) throw new IllegalStateException("Lexer factory returned null — unsupported language/model");
  return l;
}

try {
  LexerTokenizer tok = new LexerTokenizer(lexer);
} catch (IllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().contains("null Lexer")) {
    throw new TokenizerInitException("No lexer available — check language config", e);
  }
  throw e;
}

When it happens

Trigger: Calling new LexerTokenizer(null), or passing a lexer variable that a factory method returned as null (e.g. getLexer on an unsupported language/model), or a second constructor delegating a null lexer down.

Common situations: Tokenizer factory keyed by a language name not in the registry; lexer field not yet initialized before the tokenizer is built; reflection or DI wiring failing to supply the lexer dependency.

Related errors

• Cannot read from null object!
• Cannot open null document path!
• FrenchLexer: Invalid option value in constructor: :
• Invalid contraction provided to processContraction
• Not a valid dashes style:

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/ce23774d774801e4. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/LexerTokenizer.java:50

      int a = Lexer.IGNORE;
      while (a == Lexer.IGNORE) {
        a = lexer.yylex(); // skip tokens to be ignored
      }
      if (a != lexer.getYYEOF()) {
        token = lexer.yytext();
      }
      // else token remains null
    } catch (IOException e) {
      // do nothing, return null
    }
    return token;
  }

  /** Constructs a tokenizer from a {@link Lexer}.
   */
  public LexerTokenizer(Lexer l) {
    if (l == null) {
      throw new IllegalArgumentException("You can't make a Tokenizer out of a null Lexer!");
    } else {
      this.lexer = l;
    }
  }

  /** Constructs a tokenizer from a {@link Lexer} and makes a {@link Reader}
   *  the active input stream for the tokenizer.
   */
  public LexerTokenizer(Lexer l, Reader r) {
    this(l);

    try {
      l.yyreset(r);
    } catch (IOException e) {
      throw new RuntimeIOException(e.getMessage());
    }

    getNext();

View on GitHub (pinned to 1b7edd19c4)