{"record":{"id":"96a6cd9d19810383","repo":"antlr/antlr4","slug":"nexttoken-requires-a-non-null-input-stream-96a6cd","errorCode":null,"errorMessage":"nextToken requires a non-null input stream.","messagePattern":"nextToken requires a non-null input stream\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"runtime/Python3/src/antlr4/Lexer.py","lineNumber":116,"sourceCode":"        self._token = None\n        self._type = Token.INVALID_TYPE\n        self._channel = Token.DEFAULT_CHANNEL\n        self._tokenStartCharIndex = -1\n        self._tokenStartColumn = -1\n        self._tokenStartLine = -1\n        self._text = None\n\n        self._hitEOF = False\n        self._mode = Lexer.DEFAULT_MODE\n        self._modeStack = []\n\n        self._interp.reset()\n\n    # Return a token from self source; i.e., match a token on the char\n    #  stream.\n    def nextToken(self):\n        if self._input is None:\n            raise IllegalStateException(\"nextToken requires a non-null input stream.\")\n\n        # Mark start location in char stream so unbuffered streams are\n        # guaranteed at least have text of current token\n        tokenStartMarker = self._input.mark()\n        try:\n            while True:\n                if self._hitEOF:\n                    self.emitEOF()\n                    return self._token\n                self._token = None\n                self._channel = Token.DEFAULT_CHANNEL\n                self._tokenStartCharIndex = self._input.index\n                self._tokenStartColumn = self._interp.column\n                self._tokenStartLine = self._interp.line\n                self._text = None\n                continueOuter = False\n                while True:\n                    self._type = Token.INVALID_TYPE","sourceCodeStart":98,"sourceCodeEnd":134,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Python3/src/antlr4/Lexer.py#L98-L134","documentation":"Lexer.nextToken() raises IllegalStateException when self._input is None: the lexer has no character stream attached and therefore nothing to scan. The input stream is normally set via the constructor or the inputStream setter; None means the lexer was created without input, or its input was explicitly cleared (e.g., reset followed by no reassignment).","triggerScenarios":"Instantiating a generated lexer with no arguments and calling nextToken() before setting inputStream; assigning lexer.inputStream = None (the setter calls reset()) and then lexing; reusing a lexer object across files without rebinding the input between runs.","commonSituations":"Multi-file processing loops that reset the lexer but forget to attach the next InputStream; dependency-injection or test setups that construct the lexer late and call nextToken() eagerly; copy-pasted code that constructs Lexer() with parentheses as if it took no input.","solutions":["Construct the lexer with its input: lexer = MyLexer(InputStream(data))","When reusing a lexer, set lexer.inputStream = InputStream(next_data) before each nextToken()/fill() call","Add a guard: if lexer.inputStream is None: raise/skip with a clear message at the application boundary"],"exampleFix":"# before\nlexer = MyLexer()\nlexer.reset()\ntoken = lexer.nextToken()  # IllegalStateException: null input\n\n# after\nlexer = MyLexer(InputStream(data))\ntoken = lexer.nextToken()","handlingStrategy":"validation","validationCode":"# Python: verify the lexer has input attached before lexing\nif lexer.inputStream is None:\n    raise ValueError(\"attach an InputStream before calling nextToken()\")\ntoken = lexer.nextToken()","typeGuard":"# Python\ndef has_input(lexer) -> bool:\n    return getattr(lexer, \"_input\", None) is not None","tryCatchPattern":"from antlr4.error.Errors import IllegalStateException\ntry:\n    tok = lexer.nextToken()\nexcept IllegalStateException as ex:\n    if \"non-null input stream\" in str(ex):\n        lexer.inputStream = InputStream(data)  # (re)attach and retry once\n        tok = lexer.nextToken()\n    else:\n        raise","preventionTips":["Always construct lexers with an InputStream: MyLexer(InputStream(text))","In multi-file loops, reassign lexer.inputStream for each new file","Remember that setting inputStream resets lexer state — do it before, never mid-token"],"tags":["antlr","python","lexer","null-input","initialization"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}