{"record":{"id":"96f8cc0fb305e355","repo":"antlr/antlr4","slug":"tokens-cannot-be-null-96f8cc","errorCode":null,"errorMessage":"tokens cannot be null","messagePattern":"tokens cannot be null","errorType":"exception","errorClass":"ReferenceError","httpStatus":null,"severity":"error","filePath":"runtime/Python3/src/antlr4/ListTokenSource.py","lineNumber":37,"sourceCode":"\nclass ListTokenSource(TokenSource):\n    __slots__ = ('tokens', 'sourceName', 'pos', 'eofToken', '_factory')\n\n    # Constructs a new {@link ListTokenSource} instance from the specified\n    # collection of {@link Token} objects and source name.\n    #\n    # @param tokens The collection of {@link Token} objects to provide as a\n    # {@link TokenSource}.\n    # @param sourceName The name of the {@link TokenSource}. If this value is\n    # {@code null}, {@link #getSourceName} will attempt to infer the name from\n    # the next {@link Token} (or the previous token if the end of the input has\n    # been reached).\n    #\n    # @exception NullPointerException if {@code tokens} is {@code null}\n    #\n    def __init__(self, tokens:list, sourceName:str=None):\n        if tokens is None:\n            raise ReferenceError(\"tokens cannot be null\")\n        self.tokens = tokens\n        self.sourceName = sourceName\n        # The index into {@link #tokens} of token to return by the next call to\n        # {@link #nextToken}. The end of the input is indicated by this value\n        # being greater than or equal to the number of items in {@link #tokens}.\n        self.pos = 0\n        # This field caches the EOF token for the token source.\n        self.eofToken = None\n        # This is the backing field for {@link #getTokenFactory} and\n        self._factory = CommonTokenFactory.DEFAULT\n\n\n    #\n    # {@inheritDoc}\n    #\n    @property\n    def column(self):\n        if self.pos < len(self.tokens):","sourceCodeStart":19,"sourceCodeEnd":55,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Python3/src/antlr4/ListTokenSource.py#L19-L55","documentation":"ListTokenSource.__init__ raises ReferenceError('tokens cannot be null') when constructed with tokens=None. ListTokenSource wraps an existing, fully materialized token list as a TokenSource, so an actual list (possibly empty) is mandatory; None would break every subsequent nextToken() call, hence the explicit constructor guard.","triggerScenarios":"Calling ListTokenSource(None); passing a variable that was initialized to None and only conditionally assigned (e.g., 'tokens = None' then tokens = lex() only on some paths); factory code that forwards an optional parameter unchecked.","commonSituations":"Replaying recorded tokens (logging/IDE scenarios); feeding pre-lexed tokens into CommonTokenStream; optional-pipeline code where the lexing step was skipped for empty input.","solutions":["Default to an empty list: ListTokenSource(tokens or []) — an empty list is valid and immediately yields EOF","Ensure the token-producing step ran before constructing the source; fail fast if the pipeline returned None","Type-check the parameter at the call site when it crosses an API boundary"],"exampleFix":"# before\nsource = ListTokenSource(maybe_tokens)  # None when lexing was skipped\n\n# after\nsource = ListTokenSource(maybe_tokens if maybe_tokens is not None else [])","handlingStrategy":"type-guard","validationCode":"# Python: normalize the argument before construction\ntoken_list = token_list if token_list is not None else []\nsource = ListTokenSource(token_list)","typeGuard":"# Python\ndef is_token_list(value) -> bool:\n    return isinstance(value, list) and all(hasattr(t, \"tokenIndex\") for t in value)","tryCatchPattern":"try:\n    source = ListTokenSource(tokens)\nexcept ReferenceError as ex:\n    if \"tokens cannot be null\" in str(ex):\n        source = ListTokenSource([])  # empty source yields immediate EOF\n    else:\n        raise","preventionTips":["Use 'tokens or []' at the boundary when tokens may legitimately be absent","Fail fast in the lexing stage rather than letting None flow to token sources","Treat an empty token list as valid input (immediate EOF), distinct from None"],"tags":["antlr","python","token-source","null-argument","constructor"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}