{"record":{"id":"9c5468fe4cff6070","repo":"antlr/antlr4","slug":"tokensource-cannot-be-null","errorCode":null,"errorMessage":"tokenSource cannot be null","messagePattern":"tokenSource cannot be null","errorType":"exception","errorClass":"NullPointerException","httpStatus":null,"severity":"error","filePath":"runtime/Java/src/org/antlr/v4/runtime/BufferedTokenStream.java","lineNumber":71,"sourceCode":"\n\t/**\n\t * Indicates whether the {@link Token#EOF} token has been fetched from\n\t * {@link #tokenSource} and added to {@link #tokens}. This field improves\n\t * performance for the following cases:\n\t *\n\t * <ul>\n\t * <li>{@link #consume}: The lookahead check in {@link #consume} to prevent\n\t * consuming the EOF symbol is optimized by checking the values of\n\t * {@link #fetchedEOF} and {@link #p} instead of calling {@link #LA}.</li>\n\t * <li>{@link #fetch}: The check to prevent adding multiple EOF symbols into\n\t * {@link #tokens} is trivial with this field.</li>\n\t * <ul>\n\t */\n\tprotected boolean fetchedEOF;\n\n    public BufferedTokenStream(TokenSource tokenSource) {\n\t\tif (tokenSource == null) {\n\t\t\tthrow new NullPointerException(\"tokenSource cannot be null\");\n\t\t}\n        this.tokenSource = tokenSource;\n    }\n\n    @Override\n    public TokenSource getTokenSource() { return tokenSource; }\n\n\t@Override\n\tpublic int index() { return p; }\n\n    @Override\n    public int mark() {\n\t\treturn 0;\n\t}\n\n\t@Override\n\tpublic void release(int marker) {\n\t\t// no resources to release","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Java/src/org/antlr/v4/runtime/BufferedTokenStream.java#L53-L89","documentation":"BufferedTokenStream's constructor rejects a null TokenSource with a NullPointerException because the stream has no meaningful behavior without a source of tokens (every operation, from lazyInit() to fetch(), reads from tokenSource). This is a fail-fast precondition check so that misuse fails at construction instead of later during tokenization. Typical correct construction is new CommonTokenStream(lexer).","triggerScenarios":"new BufferedTokenStream(null) or new CommonTokenStream(null), usually because a Lexer variable was never assigned or a factory method returned null; passing a TokenSource field that was populated in a different lifecycle phase.","commonSituations":"Dependency injection or builder setups where the lexer is created lazily but the token stream eagerly; refactoring that moves lexer construction; testing code that stubs the lexer as null.","solutions":["Construct a Lexer first (e.g. new MyGrammarLexer(CharStreams.fromString(input))) and pass it to the token stream","Check for null lexer creation before building the stream if the lexer comes from external code","Use CharStreams.fromPath/fromString/fromStream to build the CharStream that feeds the lexer so no step returns null"],"exampleFix":"// before\nTokenStream tokens = new CommonTokenStream(null); // NPE\n\n// after\nMyGrammarLexer lexer = new MyGrammarLexer(CharStreams.fromString(input));\nTokenStream tokens = new CommonTokenStream(lexer);","handlingStrategy":"validation","validationCode":"Lexer lexer = new MyGrammarLexer(CharStreams.fromString(input));\nif (lexer == null) throw new IllegalStateException(\"lexer construction failed\");\nCommonTokenStream tokens = new CommonTokenStream(lexer);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always create the Lexer (or other TokenSource) before the token stream","Treat a null TokenSource from factories/DI as a wiring bug — fail at construction with a clear message","Prefer the idiom new CommonTokenStream(new MyLexer(CharStreams.fromX(...))) in one expression"],"tags":["antlr","token-stream","null-check","constructor"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}