{"record":{"id":"37d429c31660acd8","repo":"huggingface/tokenizers","slug":"encode-batch-inputs-can-t-be-none","errorCode":null,"errorMessage":"encode_batch: `inputs` can't be `None`","messagePattern":"encode_batch: `inputs` can't be `None`","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"bindings/python/py_src/tokenizers/implementations/base_tokenizer.py","lineNumber":258,"sourceCode":"                Each `InputSequence` can either be raw text or pre-tokenized,\n                according to the `is_pretokenized` argument:\n\n                - If `is_pretokenized=False`: `InputSequence` is expected to be `str`\n                - If `is_pretokenized=True`: `InputSequence` is expected to be\n                    `Union[List[str], Tuple[str]]`\n\n            is_pretokenized: bool:\n                Whether the input is already pre-tokenized.\n\n            add_special_tokens: bool:\n                Whether to add the special tokens while encoding.\n\n        Returns:\n            A list of Encoding\n        \"\"\"\n\n        if inputs is None:\n            raise ValueError(\"encode_batch: `inputs` can't be `None`\")\n\n        return self._tokenizer.encode_batch(inputs, is_pretokenized, add_special_tokens)\n\n    async def async_encode_batch(\n        self,\n        inputs: List[EncodeInput],\n        is_pretokenized: bool = False,\n        add_special_tokens: bool = True,\n    ) -> List[Encoding]:\n        \"\"\"Asynchronously encode a batch (tracks character offsets).\n\n        Args:\n            inputs: A list of single or pair sequences to encode.\n            is_pretokenized: Whether inputs are already pre-tokenized.\n            add_special_tokens: Whether to add special tokens.\n\n        Returns:\n            A list of Encoding.","sourceCodeStart":240,"sourceCodeEnd":276,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/py_src/tokenizers/implementations/base_tokenizer.py#L240-L276","documentation":"`Tokenizer.encode_batch` raises this `ValueError` when the `inputs` parameter is `None`. The Python wrapper performs an explicit null check before delegating to the Rust tokenizer, because `None` would otherwise surface as an opaque panic or type error inside the bindings. Passing `None` instead of an empty list or a list of sequences is almost always an upstream bug where a batch variable was never populated.","triggerScenarios":"Calling `tokenizer.encode_batch(None)` directly, or calling it with a variable that a caller/factory returned as `None` (e.g. `encode_batch(batch)` where `batch` was never initialized). Only `None` triggers it; an empty list `[]` is accepted and returns `[]`.","commonSituations":"Batching loops where `texts = get_batch()` can return `None` at the end of a dataset; config-driven pipelines where a missing dataset key defaults to `None`; passing the result of a failed file read (`open(...).read()` inside try, variable stays `None`).","solutions":["Ensure the argument is a list: pass `[]` if there is truly nothing to encode, or the list of input strings/pre-tokenized sequences.","Guard before calling: `if texts is not None: encodings = tok.encode_batch(texts)`.","Fix the upstream function that produced `None` instead of a list (e.g. return `[]` on empty input)."],"exampleFix":"// before\nencodings = tokenizer.encode_batch(batch)\n// after\nencodings = tokenizer.encode_batch(batch or [])","handlingStrategy":"validation","validationCode":"if not isinstance(inputs, list):\n    raise TypeError(f\"encode_batch expects a list, got {type(inputs).__name__}\")\nencodings = tokenizer.encode_batch(inputs or [])","typeGuard":"def is_batch(value) -> bool:\n    return isinstance(value, list)","tryCatchPattern":"try:\n    encodings = tokenizer.encode_batch(texts)\nexcept ValueError as e:\n    if \"can't be `None`\" in str(e):\n        encodings = []\n    else:\n        raise","preventionTips":["Never represent 'no data' as None in batching code; use [] consistently.","Type-hint batch producers as List[str] (never Optional) so static analysis catches None flows.","Validate batch variables at pipeline boundaries before the encode stage."],"tags":["python","null-argument","tokenizers"],"backgroundTag":"null-argument","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}