What causes "nextToken requires a non-null input stream." in antlr/antlr4?

Construct the lexer with its input: lexer = MyLexer(InputStream(data))

antlr/antlr4 · error · IllegalStateException

nextToken requires a non-null input stream.

Error message

nextToken requires a non-null input stream.

What it means

Lexer.nextToken() raises IllegalStateException when self._input is None: the lexer has no character stream attached and therefore nothing to scan. The input stream is normally set via the constructor or the inputStream setter; None means the lexer was created without input, or its input was explicitly cleared (e.g., reset followed by no reassignment).

Source

Thrown at runtime/Python3/src/antlr4/Lexer.py:116

        self._token = None
        self._type = Token.INVALID_TYPE
        self._channel = Token.DEFAULT_CHANNEL
        self._tokenStartCharIndex = -1
        self._tokenStartColumn = -1
        self._tokenStartLine = -1
        self._text = None

        self._hitEOF = False
        self._mode = Lexer.DEFAULT_MODE
        self._modeStack = []

        self._interp.reset()

    # Return a token from self source; i.e., match a token on the char
    #  stream.
    def nextToken(self):
        if self._input is None:
            raise IllegalStateException("nextToken requires a non-null input stream.")

        # Mark start location in char stream so unbuffered streams are
        # guaranteed at least have text of current token
        tokenStartMarker = self._input.mark()
        try:
            while True:
                if self._hitEOF:
                    self.emitEOF()
                    return self._token
                self._token = None
                self._channel = Token.DEFAULT_CHANNEL
                self._tokenStartCharIndex = self._input.index
                self._tokenStartColumn = self._interp.column
                self._tokenStartLine = self._interp.line
                self._text = None
                continueOuter = False
                while True:
                    self._type = Token.INVALID_TYPE

View on GitHub (pinned to 7d5770395b)

Solutions

Construct the lexer with its input: lexer = MyLexer(InputStream(data))
When reusing a lexer, set lexer.inputStream = InputStream(next_data) before each nextToken()/fill() call
Add a guard: if lexer.inputStream is None: raise/skip with a clear message at the application boundary

Example fix

# before
lexer = MyLexer()
lexer.reset()
token = lexer.nextToken()  # IllegalStateException: null input

# after
lexer = MyLexer(InputStream(data))
token = lexer.nextToken()

Defensive patterns

Strategy: validation

Validate before calling

# Python: verify the lexer has input attached before lexing
if lexer.inputStream is None:
    raise ValueError("attach an InputStream before calling nextToken()")
token = lexer.nextToken()

Type guard

# Python
def has_input(lexer) -> bool:
    return getattr(lexer, "_input", None) is not None

Try / catch

from antlr4.error.Errors import IllegalStateException
try:
    tok = lexer.nextToken()
except IllegalStateException as ex:
    if "non-null input stream" in str(ex):
        lexer.inputStream = InputStream(data)  # (re)attach and retry once
        tok = lexer.nextToken()
    else:
        raise

Prevention

Always construct lexers with an InputStream: MyLexer(InputStream(text))
In multi-file loops, reassign lexer.inputStream for each new file
Remember that setting inputStream resets lexer state — do it before, never mid-token

When it happens

Trigger: Instantiating a generated lexer with no arguments and calling nextToken() before setting inputStream; assigning lexer.inputStream = None (the setter calls reset()) and then lexing; reusing a lexer object across files without rebinding the input between runs.

Common situations: Multi-file processing loops that reset the lexer but forget to attach the next InputStream; dependency-injection or test setups that construct the lexer late and call nextToken() eagerly; copy-pasted code that constructs Lexer() with parentheses as if it took no input.

Related errors

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/96a6cd9d19810383. Report an issue: GitHub.