What causes "Invalid state number." in antlr/antlr4?

Pass only state numbers sourced from the same parser instance/ATN (e.g. ctx.invokingState of the active parse or recognizer.getATN().states lookups).

antlr/antlr4 · error · Exception

Invalid state number.

Error message

Invalid state number.

What it means

ATN.getExpectedTokens(stateNumber, ctx) validates that stateNumber addresses an existing state in the ATN (0 <= stateNumber < len(atn.states)). The ATN is the deserialized serialization of your grammar; a state number outside it means the caller passed garbage, an index from a different grammar's ATN, or a state number from an incompatible serialized ATN.

Source

Thrown at runtime/Python3/src/antlr4/atn/ATN.py:115

    # considers the complete parser context, but does not evaluate semantic
    # predicates (i.e. all predicates encountered during the calculation are
    # assumed true). If a path in the ATN exists from the starting state to the
    # {@link RuleStopState} of the outermost context without matching any
    # symbols, {@link Token#EOF} is added to the returned set.
    #
    # <p>If {@code context} is {@code null}, it is treated as
    # {@link ParserRuleContext#EMPTY}.</p>
    #
    # @param stateNumber the ATN state number
    # @param context the full parse context
    # @return The set of potentially valid input symbols which could follow the
    # specified state in the specified context.
    # @throws IllegalArgumentException if the ATN does not contain a state with
    # number {@code stateNumber}
    #/
    def getExpectedTokens(self, stateNumber:int, ctx:RuleContext ):
        if stateNumber < 0 or stateNumber >= len(self.states):
            raise Exception("Invalid state number.")
        s = self.states[stateNumber]
        following = self.nextTokens(s)
        if Token.EPSILON not in following:
            return following
        expected = IntervalSet()
        expected.addSet(following)
        expected.removeOne(Token.EPSILON)
        while (ctx != None and ctx.invokingState >= 0 and Token.EPSILON in following):
            invokingState = self.states[ctx.invokingState]
            rt = invokingState.transitions[0]
            following = self.nextTokens(rt.followState)
            expected.addSet(following)
            expected.removeOne(Token.EPSILON)
            ctx = ctx.parentCtx
        if Token.EPSILON in following:
            expected.addOne(Token.EOF)
        return expected

View on GitHub (pinned to 7d5770395b)

Solutions

Pass only state numbers sourced from the same parser instance/ATN (e.g. ctx.invokingState of the active parse or recognizer.getATN().states lookups).
Check 0 <= stateNumber < len(atn.states) before calling.
Regenerate parser/lexer with the ANTLR tool version matching the runtime, then rebuild/recopy the generated files.

Example fix

# before
atn.getExpectedTokens(9999, ctx)  # arbitrary int -> Exception

# after
atn = parser.getATN()
state = ctx.invokingState
if 0 <= state < len(atn.states):
    expected = atn.getExpectedTokens(state, ctx)

Defensive patterns

Strategy: validation

Validate before calling

atn = parser.getATN()
if 0 <= state_number < len(atn.states):
    expected = atn.getExpectedTokens(state_number, ctx)

Type guard

def is_valid_atn_state(atn, n):
    return isinstance(n, int) and 0 <= n < len(atn.states)

Try / catch

try:
    expected = atn.getExpectedTokens(state, ctx)
except Exception as e:
    if str(e) == 'Invalid state number.':
        expected = None  # degrade gracefully in diagnostics
    else:
        raise

Prevention

Only use state numbers produced by the same parser's ATN
Never hardcode state numbers; they change when the grammar changes
Keep tool and runtime versions aligned so ATN layouts match generated code

When it happens

Trigger: Calling parser.getExpectedTokens() / atn.getExpectedTokens() with a state number obtained from another parser's ATN, a hand-picked int, or after mismatched tool/runtime versions produced a misaligned ATN.

Common situations: Cross-grammar state numbers (lexer ATN state used against parser ATN); version skew between the ANTLR tool that generated the .tokens/serialized ATN and the Python runtime that deserializes it; custom error listeners that reuse ctx.invokingState from a different parse.

Related errors

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/4ff81e80653fa550. Report an issue: GitHub.