antlr/antlr4 · error · ValueError

replace: range invalid: {}..{}(size={})

Error message

replace: range invalid: {}..{}(size={})

What it means

Thrown by TokenStreamRewriter.replace() when the supplied token index range is invalid for the underlying token stream. The rewrite API validates that 0 <= from_idx <= to_idx < len(tokens.tokens) before creating a ReplaceOp; any violation raises this ValueError. It almost always means the caller computed token indices from a different token stream, used stale indices after re-lexing, or passed a Token object from another parser.

Source

Thrown at runtime/Python3/src/antlr4/TokenStreamRewriter.py:77

        rewrites = self.getProgram(program_name)
        op.instructionIndex = len(rewrites)
        rewrites.append(op)

    def replaceIndex(self, index, text):
        self.replace(self.DEFAULT_PROGRAM_NAME, index, index, text)

    def replaceRange(self, from_idx, to_idx, text):
        self.replace(self.DEFAULT_PROGRAM_NAME, from_idx, to_idx, text)

    def replaceSingleToken(self, token, text):
        self.replace(self.DEFAULT_PROGRAM_NAME, token.tokenIndex, token.tokenIndex, text)

    def replaceRangeTokens(self, from_token, to_token, text, program_name=DEFAULT_PROGRAM_NAME):
        self.replace(program_name, from_token.tokenIndex, to_token.tokenIndex, text)

    def replace(self, program_name, from_idx, to_idx, text):
        if any((from_idx > to_idx, from_idx < 0, to_idx < 0, to_idx >= len(self.tokens.tokens))):
            raise ValueError(
                'replace: range invalid: {}..{}(size={})'.format(from_idx, to_idx, len(self.tokens.tokens)))
        op = self.ReplaceOp(from_idx, to_idx, self.tokens, text)
        rewrites = self.getProgram(program_name)
        op.instructionIndex = len(rewrites)
        rewrites.append(op)

    def deleteToken(self, token):
        self.delete(self.DEFAULT_PROGRAM_NAME, token, token)

    def deleteIndex(self, index):
        self.delete(self.DEFAULT_PROGRAM_NAME, index, index)

    def delete(self, program_name, from_idx, to_idx):
        if isinstance(from_idx, Token):
            self.replace(program_name, from_idx.tokenIndex, to_idx.tokenIndex, "")
        else:
            self.replace(program_name, from_idx, to_idx, "")

View on GitHub (pinned to 7d5770395b)

Solutions

1. Recompute from_idx/to_idx directly from the same CommonTokenStream that the TokenStreamRewriter wraps, e.g. stream.getTokens(from, to) or token.tokenIndex from that stream.
2. Clamp/validate before calling: assert 0 <= from_idx <= to_idx < len(stream.tokens).
3. Prefer passing Token objects (replaceRangeTokens / deleteToken) instead of raw ints so indices come from the stream itself.
4. Remember EOF has tokenIndex == size-1; do not use to_idx == size.

Example fix

// before
rewriter.replace('default', 0, len(stream.tokens), 'x')  // to_idx past EOF -> ValueError

// after
last = len(stream.tokens) - 1  # EOF token index
rewriter.replace('default', 0, last, 'x')

def valid_replace_range(stream, from_idx, to_idx):
    return 0 <= from_idx <= to_idx < len(stream.tokens)

if valid_replace_range(stream, from_idx, to_idx):
    rewriter.replace('default', from_idx, to_idx, text)

try:
    rewriter.replace(prog, from_idx, to_idx, text)
except ValueError as e:
    if 'range invalid' in str(e):
        raise IndexError('rewrite range %d..%d outside token stream (size %d)' % (from_idx, to_idx, len(stream.tokens)))
    raise

When it happens

Trigger: Calling rewriter.replace(prog, from_idx, to_idx, text), replaceRange, replaceSingleToken, deleteIndex, insertAfter, etc. with from_idx > to_idx, a negative index, or to_idx >= size of the token stream (e.g. using token.tokenIndex values from a second parse of different input, or indexes off the end of the stream).

Common situations: Keeping Token references from one parse and applying them to a rewriter built on another CommonTokenStream; off-by-one when computing the last token index (using len(tokens) instead of len(tokens)-1); deleting/replacing the EOF token; index arithmetic done on character offsets instead of token indices.

Related errors

• replace op boundaries of {} overlap with previous {}
• insert op {} within boundaries of previous {}
• replace: range invalid: {from}..{to}(size={size})
• should be only one op per index
• Invalid state number.

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/3ba5409b2e2f0b31. Report an issue: GitHub.