antlr/antlr4 · error · NullPointerException

tokenSource cannot be null

Error message

tokenSource cannot be null

What it means

BufferedTokenStream's constructor rejects a null TokenSource with a NullPointerException because the stream has no meaningful behavior without a source of tokens (every operation, from lazyInit() to fetch(), reads from tokenSource). This is a fail-fast precondition check so that misuse fails at construction instead of later during tokenization. Typical correct construction is new CommonTokenStream(lexer).

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/BufferedTokenStream.java:71

	/**
	 * Indicates whether the {@link Token#EOF} token has been fetched from
	 * {@link #tokenSource} and added to {@link #tokens}. This field improves
	 * performance for the following cases:
	 *
	 * <ul>
	 * <li>{@link #consume}: The lookahead check in {@link #consume} to prevent
	 * consuming the EOF symbol is optimized by checking the values of
	 * {@link #fetchedEOF} and {@link #p} instead of calling {@link #LA}.</li>
	 * <li>{@link #fetch}: The check to prevent adding multiple EOF symbols into
	 * {@link #tokens} is trivial with this field.</li>
	 * <ul>
	 */
	protected boolean fetchedEOF;

    public BufferedTokenStream(TokenSource tokenSource) {
		if (tokenSource == null) {
			throw new NullPointerException("tokenSource cannot be null");
		}
        this.tokenSource = tokenSource;
    }

    @Override
    public TokenSource getTokenSource() { return tokenSource; }

	@Override
	public int index() { return p; }

    @Override
    public int mark() {
		return 0;
	}

	@Override
	public void release(int marker) {
		// no resources to release

View on GitHub (pinned to 7d5770395b)

Solutions

1. Construct a Lexer first (e.g. new MyGrammarLexer(CharStreams.fromString(input))) and pass it to the token stream
2. Check for null lexer creation before building the stream if the lexer comes from external code
3. Use CharStreams.fromPath/fromString/fromStream to build the CharStream that feeds the lexer so no step returns null

Example fix

// before
TokenStream tokens = new CommonTokenStream(null); // NPE

// after
MyGrammarLexer lexer = new MyGrammarLexer(CharStreams.fromString(input));
TokenStream tokens = new CommonTokenStream(lexer);

Lexer lexer = new MyGrammarLexer(CharStreams.fromString(input));
if (lexer == null) throw new IllegalStateException("lexer construction failed");
CommonTokenStream tokens = new CommonTokenStream(lexer);

When it happens

Trigger: new BufferedTokenStream(null) or new CommonTokenStream(null), usually because a Lexer variable was never assigned or a factory method returned null; passing a TokenSource field that was populated in a different lifecycle phase.

Common situations: Dependency injection or builder setups where the lexer is created lazily but the token stream eagerly; refactoring that moves lexer construction; testing code that stubs the lexer as null.

Related errors

• tokenSource cannot be null
• tokens cannot be null
• delegates
• tokens cannot be null
• cannot consume EOF

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/9c5468fe4cff6070. Report an issue: GitHub.