{"record":{"id":"153f7adec6db8e30","repo":"antlr/antlr4","slug":"unbuffered-stream-cannot-know-its-size-153f7a","errorCode":null,"errorMessage":"Unbuffered stream cannot know its size","messagePattern":"Unbuffered stream cannot know its size","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"warning","filePath":"runtime/Java/src/org/antlr/v4/runtime/UnbufferedTokenStream.java","lineNumber":275,"sourceCode":"\t\t}\n\t\telse if (i >= n) {\n\t\t\tthrow new UnsupportedOperationException(\"seek to index outside buffer: \"+\n\t\t\t\t\t\t\t\t\t\t\t\t\tindex+\" not in \"+ bufferStartIndex +\"..\"+(bufferStartIndex +n));\n\t\t}\n\n\t\tp = i;\n\t\tcurrentTokenIndex = index;\n\t\tif (p == 0) {\n\t\t\tlastToken = lastTokenBufferStart;\n\t\t}\n\t\telse {\n\t\t\tlastToken = tokens[p-1];\n\t\t}\n\t}\n\n\t@Override\n\tpublic int size() {\n\t\tthrow new UnsupportedOperationException(\"Unbuffered stream cannot know its size\");\n\t}\n\n\t@Override\n\tpublic String getSourceName() {\n\t\treturn tokenSource.getSourceName();\n\t}\n\n\n\t@Override\n\tpublic String getText(Interval interval) {\n\t\tint bufferStartIndex = getBufferStartIndex();\n\t\tint bufferStopIndex = bufferStartIndex + tokens.length - 1;\n\n\t\tint start = interval.a;\n\t\tint stop = interval.b;\n\t\tif (start < bufferStartIndex || stop > bufferStopIndex) {\n\t\t\tthrow new UnsupportedOperationException(\"interval \"+interval+\" not in token buffer window: \"+\n\t\t\t\t\t\t\t\t\t\t\t\t\tbufferStartIndex+\"..\"+bufferStopIndex);","sourceCodeStart":257,"sourceCodeEnd":293,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Java/src/org/antlr/v4/runtime/UnbufferedTokenStream.java#L257-L293","documentation":"UnbufferedTokenStream intentionally does not know the total token count: tokens are produced lazily and old tokens can be discarded, so IntStream.size() always throws UnsupportedOperationException. This is the documented behavior for streams whose size is unknown, not an internal failure.","triggerScenarios":"Calling size() directly on UnbufferedTokenStream, or passing it to generic code that calls TokenStream/IntStream.size() (for example rewriter or utility code that assumes a fully buffered stream).","commonSituations":"Streaming large inputs to reduce memory, then accidentally using a library routine that asks for total size; porting code from CommonTokenStream; or calling getText-like helpers that internally derive their range from size().","solutions":["Use BufferedTokenStream/CommonTokenStream when total token count is required.","Keep UnbufferedTokenStream only for one-pass streaming and track counts yourself if needed.","Guard generic code so it does not call size() for UnbufferedTokenStream instances.","Do not rely on size() after EOF; this implementation still does not retain a total count."],"exampleFix":"// before\nUnbufferedTokenStream<Token> tokens = new UnbufferedTokenStream<>(lexer);\nint count = tokens.size(); // always unsupported\n\n// after\nBufferedTokenStream tokens = new BufferedTokenStream(lexer);\nint count = tokens.size();","handlingStrategy":"type-guard","validationCode":"if (tokens instanceof UnbufferedTokenStream) {\n    throw new UnsupportedOperationException(\"Total token count requires a buffered stream\");\n}","typeGuard":"static boolean hasKnownSize(TokenStream tokens) {\n    return !(tokens instanceof UnbufferedTokenStream);\n}","tryCatchPattern":null,"preventionTips":["Pass UnbufferedTokenStream only to one-pass consumers.","Use buffered streams with APIs that require size().","Track your own token count while streaming if needed."],"tags":["antlr","java","token-stream","size","streaming"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}