antlr/antlr4 · error · NullPointerException

tokens cannot be null

Error message

tokens cannot be null

What it means

ListTokenSource's constructor throws NullPointerException when the token list is null — the source's entire purpose is to replay that list (it holds no lexer or stream). It is documented via @exception NullPointerException and is the only precondition; an empty list is legal and yields immediate EOF. Typical use is re-lexing a token subset or feeding synthesized tokens into a parser.

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/ListTokenSource.java:80

		this(tokens, null);
	}

	/**
	 * Constructs a new {@link ListTokenSource} instance from the specified
	 * collection of {@link Token} objects and source name.
	 *
	 * @param tokens The collection of {@link Token} objects to provide as a
	 * {@link TokenSource}.
	 * @param sourceName The name of the {@link TokenSource}. If this value is
	 * {@code null}, {@link #getSourceName} will attempt to infer the name from
	 * the next {@link Token} (or the previous token if the end of the input has
	 * been reached).
	 *
	 * @exception NullPointerException if {@code tokens} is {@code null}
	 */
	public ListTokenSource(List<? extends Token> tokens, String sourceName) {
		if (tokens == null) {
			throw new NullPointerException("tokens cannot be null");
		}

		this.tokens = tokens;
		this.sourceName = sourceName;
	}

	/**
	 * {@inheritDoc}
	 */
	@Override
	public int getCharPositionInLine() {
		if (i < tokens.size()) {
			return tokens.get(i).getCharPositionInLine();
		}
		else if (eofToken != null) {
			return eofToken.getCharPositionInLine();
		}
		else if (tokens.size() > 0) {

View on GitHub (pinned to 7d5770395b)

Solutions

1. Coalesce nulls to empty lists: new ListTokenSource(tokens != null ? tokens : Collections.emptyList(), name)
2. Fix the upstream getTokens(...) call to return an empty list (e.g. check the null return of get(start, stop) before use)
3. When synthesizing tokens in tests, initialize the list before constructing the source

Example fix

// before
List<Token> ts = tokens.get(0, 5); // returns null when empty
new ListTokenSource(ts, "src"); // NPE

// after
List<Token> ts = Objects.requireNonNullElseGet(tokens.get(0, 5), ArrayList::new);
new ListTokenSource(ts, "src");

List<Token> safe = (tokens != null) ? tokens : new ArrayList<>();
ListTokenSource src = new ListTokenSource(safe, name);

When it happens

Trigger: new ListTokenSource(null, name) when an upstream filtering step returned null instead of an empty list; building token sources from optional collections without defaults; map/filter pipelines that propagate null.

Common situations: Token-rewriting pipelines (e.g. dropping comment tokens then reparsing); code using getTokens(...) overloads that return null for empty results and passing that straight to ListTokenSource.

Related errors

• tokens cannot be null
• tokenSource cannot be null
• delegates
• tokenSource cannot be null
• tokens cannot be null

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/61d0631dc8bed868. Report an issue: GitHub.