antlr/antlr4 · error · Exception

${tokenIndex} not in 0..${len(tokens)-1}

Error message

${tokenIndex} not in 0..${len(tokens)-1}

What it means

BufferedTokenStream.getHiddenTokensToRight(tokenIndex) validates that tokenIndex falls within the currently fetched token list and raises a generic Exception ('<n> not in 0..<len-1>') otherwise. The index is an absolute position in the token buffer, and lazyInit must have run, so the error means the index refers to a token that has not been fetched (or a negative/garbage value).

Source

Thrown at runtime/Python3/src/antlr4/BufferedTokenStream.py:232

            self.sync(i)
            token = self.tokens[i]
        return i

    # Given a starting index, return the index of the previous token on channel.
    #  Return i if tokens[i] is on channel. Return -1 if there are no tokens
    #  on channel between i and 0.
    def previousTokenOnChannel(self, i:int, channel:int):
        while i>=0 and self.tokens[i].channel!=channel:
            i -= 1
        return i

    # Collect all tokens on specified channel to the right of
    #  the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or
    #  EOF. If channel is -1, find any non default channel token.
    def getHiddenTokensToRight(self, tokenIndex:int, channel:int=-1):
        self.lazyInit()
        if tokenIndex<0 or tokenIndex>=len(self.tokens):
            raise Exception(str(tokenIndex) + " not in 0.." + str(len(self.tokens)-1))
        from .Lexer import Lexer
        nextOnChannel = self.nextTokenOnChannel(tokenIndex + 1, Lexer.DEFAULT_TOKEN_CHANNEL)
        from_ = tokenIndex+1
        # if none onchannel to right, nextOnChannel=-1 so set to = last token
        to = (len(self.tokens)-1) if nextOnChannel==-1 else nextOnChannel
        return self.filterForChannel(from_, to, channel)


    # Collect all tokens on specified channel to the left of
    #  the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.
    #  If channel is -1, find any non default channel token.
    def getHiddenTokensToLeft(self, tokenIndex:int, channel:int=-1):
        self.lazyInit()
        if tokenIndex<0 or tokenIndex>=len(self.tokens):
            raise Exception(str(tokenIndex) + " not in 0.." + str(len(self.tokens)-1))
        from .Lexer import Lexer
        prevOnChannel = self.previousTokenOnChannel(tokenIndex - 1, Lexer.DEFAULT_TOKEN_CHANNEL)
        if prevOnChannel == tokenIndex - 1:

View on GitHub (pinned to 7d5770395b)

Solutions

1. Bound-check first: 0 <= tokenIndex < len(tokens) using the stream's own list (e.g., after tokens.fill() for CommonTokenStream)
2. Call tokens.fill() (buffered stream) before indexing so the full token list exists
3. Pass token.getTokenIndex() from tokens produced by the same stream, never raw offsets

Example fix

# before
hidden = tokens.getHiddenTokensToRight(idx)  # idx may be out of range

# after
if 0 <= idx < len(tokens.tokens):
    hidden = tokens.getHiddenTokensToRight(idx)
else:
    hidden = None

# Python: bound-check against the stream's own token list
if 0 <= token_index < len(tokens.tokens):
    hidden = tokens.getHiddenTokensToRight(token_index)
else:
    hidden = None

try:
    hidden = tokens.getHiddenTokensToRight(token_index)
except Exception as ex:
    if " not in 0.." in str(ex):
        hidden = None  # index outside fetched tokens; treat as no hidden tokens
    else:
        raise

When it happens

Trigger: tokenIndex < 0 or tokenIndex >= len(self.tokens): passing a token's getTokenIndex() from a different stream, passing a character offset instead of a token index, or calling before the stream has lexed up to that point (lazyInit only fetches the first token).

Common situations: Comment/whitespace extraction utilities that walk tokens and call getHiddenTokensToRight on the last token before EOF was fetched; mixing up lexer char indices and token indices; using a token list built by a separate pass with different indexing.

Related errors

• {tokenIndex} not in 0..{tokens.Count - 1}
• {} not in 0..{}
• token index {i} out of range 0..{tokens.Count - 1}
• start {start} or stop {stop} not in 0..{tokens.Count - 1}
• cannot consume EOF

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/15f63e0716572a6b. Report an issue: GitHub.