antlr/antlr4 · error · IndexOutOfBoundsException

get({i}) outside buffer: {bufferStartIndex}..{bufferStartInd

Error message

get({i}) outside buffer: {bufferStartIndex}..{bufferStartIndex+n}

What it means

UnbufferedTokenStream.get(i) resolves absolute token indexes against the retained buffer window; if i is before bufferStartIndex (already discarded) or at/after bufferStartIndex + n (not yet buffered), it throws IndexOutOfBoundsException. The unbuffered token stream only supports lookaround within its window.

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/UnbufferedTokenStream.java:82

	 */
	protected int currentTokenIndex = 0;

	public UnbufferedTokenStream(TokenSource tokenSource) {
		this(tokenSource, 256);
	}

	public UnbufferedTokenStream(TokenSource tokenSource, int bufferSize) {
		this.tokenSource = tokenSource;
		tokens = new Token[bufferSize];
		n = 0;
		fill(1); // prime the pump
	}

	@Override
	public Token get(int i) { // get absolute index
		int bufferStartIndex = getBufferStartIndex();
		if (i < bufferStartIndex || i >= bufferStartIndex + n) {
			throw new IndexOutOfBoundsException("get("+i+") outside buffer: "+
			                    bufferStartIndex+".."+(bufferStartIndex+n));
		}
		return tokens[i - bufferStartIndex];
	}

	@Override
	public Token LT(int i) {
		if ( i==-1 ) {
			return lastToken;
		}

		sync(i);
        int index = p + i - 1;
        if ( index < 0 ) {
			throw new IndexOutOfBoundsException("LT("+i+") gives negative index");
		}

		if ( index >= n ) {

View on GitHub (pinned to 7d5770395b)

Solutions

1. Use CommonTokenStream (buffered) when random access to all tokens is required
2. With the unbuffered stream, access tokens only via LT/LA relative to the current index, or hold marks to pin the window
3. Check the window bounds before calling get(): bufferStartIndex <= i < bufferStartIndex + n (expose bufferStartIndex as index() - p, or track via marks)

Example fix

// before
TokenStream tokens = new UnbufferedTokenStream(lexer);
// after parsing:
Token first = tokens.get(0); // IndexOutOfBoundsException: already discarded

// after
TokenStream tokens = new CommonTokenStream(lexer); // buffers all tokens
Token first = tokens.get(0);

int bufferStart = tokens.index() - /* p not exposed: track a floor via marks */ floorTokens;
if (i >= bufferStart && i < bufferStart + bufferedCount) {
    Token t = tokens.get(i);
} else {
    /* out of window: use cached tokens or a buffered stream */
}

static boolean supportsRandomAccess(TokenStream ts) {
    return !(ts instanceof UnbufferedTokenStream);
}

try { tokens.get(i); }
catch (IndexOutOfBoundsException e) { /* token discarded: re-parse with CommonTokenStream */ }

When it happens

Trigger: get(i) for a token index far behind the current position after markers were released; get(i) ahead of the buffer without a preceding sync/fill; utilities that index tokens by absolute position (e.g., building a token list for an IDE).

Common situations: Error listeners or post-processing that fetch arbitrary historical tokens; porting code written against CommonTokenStream (which buffers everything) to UnbufferedTokenStream; token-annotating tools that assume random access.

Related errors

• token index {} out of range 0..{}
• start {} or stop {} not in 0..{}
• {} not in 0..{}
• LT({i}) gives negative index
• seek to index outside buffer: {index} not in {bufferStartInd

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/e5a32e984da1f2b8. Report an issue: GitHub.