{"record":{"id":"15f63e0716572a6b","repo":"antlr/antlr4","slug":"tokenindex-not-in-0-len-tokens-1","errorCode":null,"errorMessage":"${tokenIndex} not in 0..${len(tokens)-1}","messagePattern":"(.+?) not in 0\\.\\.(.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"runtime/Python3/src/antlr4/BufferedTokenStream.py","lineNumber":232,"sourceCode":"            self.sync(i)\n            token = self.tokens[i]\n        return i\n\n    # Given a starting index, return the index of the previous token on channel.\n    #  Return i if tokens[i] is on channel. Return -1 if there are no tokens\n    #  on channel between i and 0.\n    def previousTokenOnChannel(self, i:int, channel:int):\n        while i>=0 and self.tokens[i].channel!=channel:\n            i -= 1\n        return i\n\n    # Collect all tokens on specified channel to the right of\n    #  the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or\n    #  EOF. If channel is -1, find any non default channel token.\n    def getHiddenTokensToRight(self, tokenIndex:int, channel:int=-1):\n        self.lazyInit()\n        if tokenIndex<0 or tokenIndex>=len(self.tokens):\n            raise Exception(str(tokenIndex) + \" not in 0..\" + str(len(self.tokens)-1))\n        from .Lexer import Lexer\n        nextOnChannel = self.nextTokenOnChannel(tokenIndex + 1, Lexer.DEFAULT_TOKEN_CHANNEL)\n        from_ = tokenIndex+1\n        # if none onchannel to right, nextOnChannel=-1 so set to = last token\n        to = (len(self.tokens)-1) if nextOnChannel==-1 else nextOnChannel\n        return self.filterForChannel(from_, to, channel)\n\n\n    # Collect all tokens on specified channel to the left of\n    #  the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.\n    #  If channel is -1, find any non default channel token.\n    def getHiddenTokensToLeft(self, tokenIndex:int, channel:int=-1):\n        self.lazyInit()\n        if tokenIndex<0 or tokenIndex>=len(self.tokens):\n            raise Exception(str(tokenIndex) + \" not in 0..\" + str(len(self.tokens)-1))\n        from .Lexer import Lexer\n        prevOnChannel = self.previousTokenOnChannel(tokenIndex - 1, Lexer.DEFAULT_TOKEN_CHANNEL)\n        if prevOnChannel == tokenIndex - 1:","sourceCodeStart":214,"sourceCodeEnd":250,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Python3/src/antlr4/BufferedTokenStream.py#L214-L250","documentation":"BufferedTokenStream.getHiddenTokensToRight(tokenIndex) validates that tokenIndex falls within the currently fetched token list and raises a generic Exception ('<n> not in 0..<len-1>') otherwise. The index is an absolute position in the token buffer, and lazyInit must have run, so the error means the index refers to a token that has not been fetched (or a negative/garbage value).","triggerScenarios":"tokenIndex < 0 or tokenIndex >= len(self.tokens): passing a token's getTokenIndex() from a different stream, passing a character offset instead of a token index, or calling before the stream has lexed up to that point (lazyInit only fetches the first token).","commonSituations":"Comment/whitespace extraction utilities that walk tokens and call getHiddenTokensToRight on the last token before EOF was fetched; mixing up lexer char indices and token indices; using a token list built by a separate pass with different indexing.","solutions":["Bound-check first: 0 <= tokenIndex < len(tokens) using the stream's own list (e.g., after tokens.fill() for CommonTokenStream)","Call tokens.fill() (buffered stream) before indexing so the full token list exists","Pass token.getTokenIndex() from tokens produced by the same stream, never raw offsets"],"exampleFix":"# before\nhidden = tokens.getHiddenTokensToRight(idx)  # idx may be out of range\n\n# after\nif 0 <= idx < len(tokens.tokens):\n    hidden = tokens.getHiddenTokensToRight(idx)\nelse:\n    hidden = None","handlingStrategy":"validation","validationCode":"# Python: bound-check against the stream's own token list\nif 0 <= token_index < len(tokens.tokens):\n    hidden = tokens.getHiddenTokensToRight(token_index)\nelse:\n    hidden = None","typeGuard":null,"tryCatchPattern":"try:\n    hidden = tokens.getHiddenTokensToRight(token_index)\nexcept Exception as ex:\n    if \" not in 0..\" in str(ex):\n        hidden = None  # index outside fetched tokens; treat as no hidden tokens\n    else:\n        raise","preventionTips":["Call tokens.fill() on CommonTokenStream before doing index math on hidden tokens","Always derive indices via token.tokenIndex from the same stream","Distinguish char offsets from token indices in your variable naming to avoid mix-ups"],"tags":["antlr","python","token-stream","hidden-tokens","index-out-of-range"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}