{"record":{"id":"ce23774d774801e4","repo":"stanfordnlp/CoreNLP","slug":"you-can-t-make-a-tokenizer-out-of-a-null-lexer","errorCode":null,"errorMessage":"You can't make a Tokenizer out of a null Lexer!","messagePattern":"You can't make a Tokenizer out of a null Lexer!","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/process/LexerTokenizer.java","lineNumber":50,"sourceCode":"      int a = Lexer.IGNORE;\n      while (a == Lexer.IGNORE) {\n        a = lexer.yylex(); // skip tokens to be ignored\n      }\n      if (a != lexer.getYYEOF()) {\n        token = lexer.yytext();\n      }\n      // else token remains null\n    } catch (IOException e) {\n      // do nothing, return null\n    }\n    return token;\n  }\n\n  /** Constructs a tokenizer from a {@link Lexer}.\n   */\n  public LexerTokenizer(Lexer l) {\n    if (l == null) {\n      throw new IllegalArgumentException(\"You can't make a Tokenizer out of a null Lexer!\");\n    } else {\n      this.lexer = l;\n    }\n  }\n\n  /** Constructs a tokenizer from a {@link Lexer} and makes a {@link Reader}\n   *  the active input stream for the tokenizer.\n   */\n  public LexerTokenizer(Lexer l, Reader r) {\n    this(l);\n\n    try {\n      l.yyreset(r);\n    } catch (IOException e) {\n      throw new RuntimeIOException(e.getMessage());\n    }\n\n    getNext();","sourceCodeStart":32,"sourceCodeEnd":68,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/process/LexerTokenizer.java#L32-L68","documentation":"LexerTokenizer wraps a JFlex-generated Lexer; constructing it with a null Lexer would make every token request fail, so the constructor throws this IllegalArgumentException. It is a simple fail-fast guard on the mandatory lexer argument.","triggerScenarios":"Calling new LexerTokenizer(null), or passing a lexer variable that a factory method returned as null (e.g. getLexer on an unsupported language/model), or a second constructor delegating a null lexer down.","commonSituations":"Tokenizer factory keyed by a language name not in the registry; lexer field not yet initialized before the tokenizer is built; reflection or DI wiring failing to supply the lexer dependency.","solutions":["Check why the Lexer was null: trace the factory/lookup that should have created it","Construct the lexer explicitly, e.g. new LexerTokenizer(new PTBLexer(new InputStreamReader(in)))","Verify the language/tokenizer option string maps to an available lexer class on the classpath","Add an assertion/null-check at the call site to fail with a more descriptive message"],"exampleFix":"// before\nLexer lex = lexerFactory.get(lang); // may be null\nLexerTokenizer tok = new LexerTokenizer(lex);\n// after\nLexer lex = lexerFactory.get(lang);\nif (lex == null) {\n  throw new IllegalStateException(\"No lexer registered for language: \" + lang);\n}\nLexerTokenizer tok = new LexerTokenizer(lex);","handlingStrategy":"type-guard","validationCode":"Objects.requireNonNull(lexer, \"Lexer must be created before building LexerTokenizer\");","typeGuard":"static Lexer requireLexer(Lexer l) {\n  if (l == null) throw new IllegalStateException(\"Lexer factory returned null — unsupported language/model\");\n  return l;\n}","tryCatchPattern":"try {\n  LexerTokenizer tok = new LexerTokenizer(lexer);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"null Lexer\")) {\n    throw new TokenizerInitException(\"No lexer available — check language config\", e);\n  }\n  throw e;\n}","preventionTips":["Verify the language/tokenizer option maps to a real lexer before constructing","Check factory results for null immediately and fail with a descriptive message","Cover tokenizer construction in unit tests for every configured language"],"tags":["null","constructor","tokenizer","illegal-argument","precondition"],"backgroundTag":"null-argument","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}