antlr/antlr4 · error · UnsupportedOperationException

interval {interval} not in token buffer window: {bufferStart

Error message

interval {interval} not in token buffer window: {bufferStartIndex}..{bufferStopIndex}

What it means

UnbufferedTokenStream.getText(Interval) can return text only from tokens still present in its allocated buffer window. It rejects intervals whose start is before bufferStartIndex or whose stop is beyond bufferStartIndex + tokens.length - 1. Once old tokens are flushed, their text cannot be recovered.

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/UnbufferedTokenStream.java:292

	public int size() {
		throw new UnsupportedOperationException("Unbuffered stream cannot know its size");
	}

	@Override
	public String getSourceName() {
		return tokenSource.getSourceName();
	}


	@Override
	public String getText(Interval interval) {
		int bufferStartIndex = getBufferStartIndex();
		int bufferStopIndex = bufferStartIndex + tokens.length - 1;

		int start = interval.a;
		int stop = interval.b;
		if (start < bufferStartIndex || stop > bufferStopIndex) {
			throw new UnsupportedOperationException("interval "+interval+" not in token buffer window: "+
													bufferStartIndex+".."+bufferStopIndex);
		}

		int a = start - bufferStartIndex;
		int b = stop - bufferStartIndex;

		StringBuilder buf = new StringBuilder();
		for (int i = a; i <= b; i++) {
			Token t = tokens[i];
			buf.append(t.getText());
		}

		return buf.toString();
	}

	protected final int getBufferStartIndex() {
		return currentTokenIndex - p;
	}

View on GitHub (pinned to 7d5770395b)

Solutions

1. Use CommonTokenStream/BufferedTokenStream when parse-tree text extraction is required.
2. Call getText while the interval is still buffered and keep a mark active during lookahead/backtracking.
3. Load enough lookahead before requesting future intervals.
4. Recreate the token stream if text from already-discarded tokens is required.

Example fix

// before
UnbufferedTokenStream<Token> tokens = new UnbufferedTokenStream<>(lexer);
parser.setTokenStream(tokens);
ParseTree tree = parser.start();
String text = tokens.getText(tree.getSourceInterval()); // earlier tokens may be gone

// after
CommonTokenStream tokens = new CommonTokenStream(lexer);
parser.setTokenStream(tokens);
ParseTree tree = parser.start();
String text = tokens.getText(tree.getSourceInterval());

static boolean intervalLooksRetained(UnbufferedTokenStream<Token> tokens, Interval interval) {
    if (interval.a < 0 || interval.b < interval.a) return false;
    int current = tokens.index();
    if (interval.b >= current) {
        tokens.LT(interval.b - current + 1); // load ahead, clamps to EOF
    }
    try {
        tokens.get(interval.a);
        tokens.get(interval.b);
        return true;
    } catch (IndexOutOfBoundsException e) {
        return false;
    }
}

static boolean supportsArbitraryGetText(TokenStream tokens) {
    return tokens instanceof BufferedTokenStream;
}

try {
    return tokens.getText(interval);
} catch (UnsupportedOperationException e) {
    // requested interval left the buffer; retain a buffer or re-lex the source
}

When it happens

Trigger: Calling getText(ctx.getSourceInterval()), getText(Token,Token), or getText(Interval) after earlier tokens were flushed without a marker; requesting a stop token that has not been buffered; or passing an interval from a parse tree whose tokens precede the current buffer start.

Common situations: Extracting text from a parse tree after parsing completed, using UnbufferedTokenStream to save memory, and generic reporting/rewriting utilities that request arbitrary intervals.

Related errors

• interval {interval} outside buffer: {bufferStartIndex}..{buf
• get({i}) outside buffer: {bufferStartIndex}..{bufferStartInd
• LT({i}) gives negative index
• cannot consume EOF
• release() called with an invalid marker.

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/6d44125c75d34fd2. Report an issue: GitHub.