{"record":{"id":"8ba7643eeb403c9e","repo":"antlr/antlr4","slug":"nexttoken-requires-a-non-null-input-stream","errorCode":null,"errorMessage":"nextToken requires a non-null input stream.","messagePattern":"nextToken requires a non-null input stream\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"runtime/Java/src/org/antlr/v4/runtime/Lexer.java","lineNumber":114,"sourceCode":"\t\t_tokenStartCharIndex = -1;\n\t\t_tokenStartCharPositionInLine = -1;\n\t\t_tokenStartLine = -1;\n\t\t_text = null;\n\n\t\t_hitEOF = false;\n\t\t_mode = Lexer.DEFAULT_MODE;\n\t\t_modeStack.clear();\n\n\t\tgetInterpreter().reset();\n\t}\n\n\t/** Return a token from this source; i.e., match a token on the char\n\t *  stream.\n\t */\n\t@Override\n\tpublic Token nextToken() {\n\t\tif (_input == null) {\n\t\t\tthrow new IllegalStateException(\"nextToken requires a non-null input stream.\");\n\t\t}\n\n\t\t// Mark start location in char stream so unbuffered streams are\n\t\t// guaranteed at least have text of current token\n\t\tint tokenStartMarker = _input.mark();\n\t\ttry{\n\t\t\touter:\n\t\t\twhile (true) {\n\t\t\t\tif (_hitEOF) {\n\t\t\t\t\temitEOF();\n\t\t\t\t\treturn _token;\n\t\t\t\t}\n\n\t\t\t\t_token = null;\n\t\t\t\t_channel = Token.DEFAULT_CHANNEL;\n\t\t\t\t_tokenStartCharIndex = _input.index();\n\t\t\t\t_tokenStartCharPositionInLine = getInterpreter().getCharPositionInLine();\n\t\t\t\t_tokenStartLine = getInterpreter().getLine();","sourceCodeStart":96,"sourceCodeEnd":132,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Java/src/org/antlr/v4/runtime/Lexer.java#L96-L132","documentation":"Lexer.nextToken() throws IllegalStateException when the lexer's _input CharStream is null — matching requires reading characters, so there is nothing sensible to return. The lexer normally receives its CharStream via setInputStream() or the generated constructor; a null input means the lexer was constructed or reset incorrectly (e.g. reused after being passed null).","triggerScenarios":"Constructing a generated Lexer with the no-arg (or null) constructor and forgetting setInputStream(CharStream); calling reset() then nextToken() on a lexer whose input was never set; DI frameworks instantiating the lexer without wiring the stream.","commonSituations":"Reusing a single Lexer object across files and forgetting setInputStream() between parses; testing harnesses that new the lexer directly; refactoring that removed the constructor argument.","solutions":["Always set the input before tokenizing: lexer.setInputStream(CharStreams.fromString(input)) or construct with the stream","Create a fresh Lexer per input (recommended; they are cheap) rather than reusing across files","Null-check the CharStream-producing code path (file read, URL fetch) before assigning it to the lexer"],"exampleFix":"// before\nMyLexer lexer = new MyLexer(null); // later nextToken() throws\n\n// after\nMyLexer lexer = new MyLexer(CharStreams.fromString(input));\n// or: lexer.setInputStream(CharStreams.fromPath(path));","handlingStrategy":"validation","validationCode":"CharStream input = CharStreams.fromString(text);\nif (input != null) {\n  lexer.setInputStream(input);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always construct the Lexer with a CharStream or call setInputStream before tokenizing","Create a fresh Lexer per input instead of reusing across files","Null-check CharStream producers (file loaders, HTTP fetches) at their source"],"tags":["antlr","lexer","null-input","lifecycle"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}