antlr/antlr4 · error · NotSupportedException

Unbuffered stream cannot know its size

Error message

Unbuffered stream cannot know its size

What it means

UnbufferedTokenStream.Size deliberately throws NotSupportedException because the stream lexes lazily and discards consumed tokens, so the total token count is unknowable without consuming everything. Any code path that reads the Size property on this stream type will fail.

Source

Thrown at runtime/CSharp/src/UnbufferedTokenStream.cs:366

                }
            }
            p = i;
            currentTokenIndex = index;
            if (p == 0)
            {
                lastToken = lastTokenBufferStart;
            }
            else
            {
                lastToken = tokens[p - 1];
            }
        }

        public virtual int Size
        {
            get
            {
                throw new NotSupportedException("Unbuffered stream cannot know its size");
            }
        }

        public virtual string SourceName
        {
            get
            {
                return TokenSource.SourceName;
            }
        }

        [return: NotNull]
        public virtual string GetText(Interval interval)
        {
            int bufferStartIndex = GetBufferStartIndex();
            int bufferStopIndex = bufferStartIndex + tokens.Length - 1;
            int start = interval.a;
            int stop = interval.b;

View on GitHub (pinned to 7d5770395b)

Solutions

1. Use CommonTokenStream instead when token count or full random access is needed
2. If only a count is required and input is finite, wrap a BufferedTokenStream over the same token source and read its Size
3. Count tokens manually by consuming to EOF on a separate pass if memory constraints still forbid buffering

Example fix

// before
var tokens = new UnbufferedTokenStream(lexer);
int n = tokens.Size; // throws NotSupportedException

// after
var tokens = new CommonTokenStream(lexer);
int n = tokens.Size; // works: buffered stream knows its size

// C#: probe capability instead of catching
bool knowsSize = stream is BufferedTokenStream; // CommonTokenStream derives from it
int size = knowsSize ? ((BufferedTokenStream)stream).Size : CountByConsuming(stream);

// C#
static bool KnowsSize(Antlr4.Runtime.ITokenStream ts) => ts is Antlr4.Runtime.BufferedTokenStream;

try { int n = stream.Size; } catch (NotSupportedException) { // unbuffered stream: obtain count by a full consuming pass or switch to CommonTokenStream }

When it happens

Trigger: Accessing stream.Size on an UnbufferedTokenStream directly; passing the unbuffered stream to helper code (pretty-printers, error reporters, profilers, tree pattern machinery) that queries Size; calling APIs that preallocate based on token count.

Common situations: Swapping CommonTokenStream for UnbufferedTokenStream to reduce memory on huge inputs, then hitting library/utility code that assumes Size exists; unit-test harnesses that assert on token counts; migration from buffered to unbuffered streaming.

Related errors

• get(${i}) outside buffer: ${bufferStartIndex}..${bufferStart
• cannot consume EOF
• Unbuffered stream cannot know its size
• tokenSource cannot be null
• cannot consume EOF

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/ccb3379888843456. Report an issue: GitHub.