antlr/antlr4 · warning · UnsupportedOperationException

Unbuffered stream cannot know its size

Error message

Unbuffered stream cannot know its size

What it means

UnbufferedTokenStream intentionally does not know the total token count: tokens are produced lazily and old tokens can be discarded, so IntStream.size() always throws UnsupportedOperationException. This is the documented behavior for streams whose size is unknown, not an internal failure.

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/UnbufferedTokenStream.java:275

		}
		else if (i >= n) {
			throw new UnsupportedOperationException("seek to index outside buffer: "+
													index+" not in "+ bufferStartIndex +".."+(bufferStartIndex +n));
		}

		p = i;
		currentTokenIndex = index;
		if (p == 0) {
			lastToken = lastTokenBufferStart;
		}
		else {
			lastToken = tokens[p-1];
		}
	}

	@Override
	public int size() {
		throw new UnsupportedOperationException("Unbuffered stream cannot know its size");
	}

	@Override
	public String getSourceName() {
		return tokenSource.getSourceName();
	}


	@Override
	public String getText(Interval interval) {
		int bufferStartIndex = getBufferStartIndex();
		int bufferStopIndex = bufferStartIndex + tokens.length - 1;

		int start = interval.a;
		int stop = interval.b;
		if (start < bufferStartIndex || stop > bufferStopIndex) {
			throw new UnsupportedOperationException("interval "+interval+" not in token buffer window: "+
													bufferStartIndex+".."+bufferStopIndex);

View on GitHub (pinned to 7d5770395b)

Solutions

1. Use BufferedTokenStream/CommonTokenStream when total token count is required.
2. Keep UnbufferedTokenStream only for one-pass streaming and track counts yourself if needed.
3. Guard generic code so it does not call size() for UnbufferedTokenStream instances.
4. Do not rely on size() after EOF; this implementation still does not retain a total count.

Example fix

// before
UnbufferedTokenStream<Token> tokens = new UnbufferedTokenStream<>(lexer);
int count = tokens.size(); // always unsupported

// after
BufferedTokenStream tokens = new BufferedTokenStream(lexer);
int count = tokens.size();

if (tokens instanceof UnbufferedTokenStream) {
    throw new UnsupportedOperationException("Total token count requires a buffered stream");
}

static boolean hasKnownSize(TokenStream tokens) {
    return !(tokens instanceof UnbufferedTokenStream);
}

When it happens

Trigger: Calling size() directly on UnbufferedTokenStream, or passing it to generic code that calls TokenStream/IntStream.size() (for example rewriter or utility code that assumes a fully buffered stream).

Common situations: Streaming large inputs to reduce memory, then accidentally using a library routine that asks for total size; porting code from CommonTokenStream; or calling getText-like helpers that internally derive their range from size().

Related errors

• cannot consume EOF
• release() called with an invalid marker.
• cannot seek to negative index {index}
• seek to index outside buffer: {index} not in {bufferStartInd
• interval {interval} not in token buffer window: {bufferStart

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/153f7adec6db8e30. Report an issue: GitHub.