antlr/antlr4 · error · ReferenceError

tokens cannot be null

Error message

tokens cannot be null

What it means

ListTokenSource.__init__ raises ReferenceError('tokens cannot be null') when constructed with tokens=None. ListTokenSource wraps an existing, fully materialized token list as a TokenSource, so an actual list (possibly empty) is mandatory; None would break every subsequent nextToken() call, hence the explicit constructor guard.

Source

Thrown at runtime/Python3/src/antlr4/ListTokenSource.py:37

class ListTokenSource(TokenSource):
    __slots__ = ('tokens', 'sourceName', 'pos', 'eofToken', '_factory')

    # Constructs a new {@link ListTokenSource} instance from the specified
    # collection of {@link Token} objects and source name.
    #
    # @param tokens The collection of {@link Token} objects to provide as a
    # {@link TokenSource}.
    # @param sourceName The name of the {@link TokenSource}. If this value is
    # {@code null}, {@link #getSourceName} will attempt to infer the name from
    # the next {@link Token} (or the previous token if the end of the input has
    # been reached).
    #
    # @exception NullPointerException if {@code tokens} is {@code null}
    #
    def __init__(self, tokens:list, sourceName:str=None):
        if tokens is None:
            raise ReferenceError("tokens cannot be null")
        self.tokens = tokens
        self.sourceName = sourceName
        # The index into {@link #tokens} of token to return by the next call to
        # {@link #nextToken}. The end of the input is indicated by this value
        # being greater than or equal to the number of items in {@link #tokens}.
        self.pos = 0
        # This field caches the EOF token for the token source.
        self.eofToken = None
        # This is the backing field for {@link #getTokenFactory} and
        self._factory = CommonTokenFactory.DEFAULT


    #
    # {@inheritDoc}
    #
    @property
    def column(self):
        if self.pos < len(self.tokens):

View on GitHub (pinned to 7d5770395b)

Solutions

1. Default to an empty list: ListTokenSource(tokens or []) — an empty list is valid and immediately yields EOF
2. Ensure the token-producing step ran before constructing the source; fail fast if the pipeline returned None
3. Type-check the parameter at the call site when it crosses an API boundary

Example fix

# before
source = ListTokenSource(maybe_tokens)  # None when lexing was skipped

# after
source = ListTokenSource(maybe_tokens if maybe_tokens is not None else [])

# Python: normalize the argument before construction
token_list = token_list if token_list is not None else []
source = ListTokenSource(token_list)

# Python
def is_token_list(value) -> bool:
    return isinstance(value, list) and all(hasattr(t, "tokenIndex") for t in value)

try:
    source = ListTokenSource(tokens)
except ReferenceError as ex:
    if "tokens cannot be null" in str(ex):
        source = ListTokenSource([])  # empty source yields immediate EOF
    else:
        raise

When it happens

Trigger: Calling ListTokenSource(None); passing a variable that was initialized to None and only conditionally assigned (e.g., 'tokens = None' then tokens = lex() only on some paths); factory code that forwards an optional parameter unchecked.

Common situations: Replaying recorded tokens (logging/IDE scenarios); feeding pre-lexed tokens into CommonTokenStream; optional-pipeline code where the lexing step was skipped for empty input.

Related errors

• tokens cannot be null
• tokens cannot be null
• listener
• missing interface implementation
• Fatal error occured while evaluating the names of the gramma

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/96f8cc0fb305e355. Report an issue: GitHub.