{"record":{"id":"af7b06b6a2e608e5","repo":"huggingface/tokenizers","slug":"async-encode-batch-inputs-can-t-be-none","errorCode":null,"errorMessage":"async_encode_batch: `inputs` can't be `None`","messagePattern":"async_encode_batch: `inputs` can't be `None`","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"bindings/python/py_src/tokenizers/implementations/base_tokenizer.py","lineNumber":279,"sourceCode":"\n    async def async_encode_batch(\n        self,\n        inputs: List[EncodeInput],\n        is_pretokenized: bool = False,\n        add_special_tokens: bool = True,\n    ) -> List[Encoding]:\n        \"\"\"Asynchronously encode a batch (tracks character offsets).\n\n        Args:\n            inputs: A list of single or pair sequences to encode.\n            is_pretokenized: Whether inputs are already pre-tokenized.\n            add_special_tokens: Whether to add special tokens.\n\n        Returns:\n            A list of Encoding.\n        \"\"\"\n        if inputs is None:\n            raise ValueError(\"async_encode_batch: `inputs` can't be `None`\")\n        # Exposed by the Rust bindings via pyo3_async_runtimes::tokio::future_into_py\n        return await self._tokenizer.async_encode_batch(inputs, is_pretokenized, add_special_tokens)\n\n    async def async_encode_batch_fast(\n        self,\n        inputs: List[EncodeInput],\n        is_pretokenized: bool = False,\n        add_special_tokens: bool = True,\n    ) -> List[Encoding]:\n        \"\"\"Asynchronously encode a batch (no character offsets, faster).\n\n        Args:\n            inputs: A list of single or pair sequences to encode.\n            is_pretokenized: Whether inputs are already pre-tokenized.\n            add_special_tokens: Whether to add special tokens.\n\n        Returns:\n            A list of Encoding.","sourceCodeStart":261,"sourceCodeEnd":297,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/py_src/tokenizers/implementations/base_tokenizer.py#L261-L297","documentation":"`Tokenizer.async_encode_batch` raises this `ValueError` when `inputs` is `None`. As with the sync variant, the Python layer validates the argument before awaiting the Rust implementation exposed via pyo3, so a null batch is rejected with a clear message rather than a binding-level failure.","triggerScenarios":"Awaiting `await tokenizer.async_encode_batch(None)`, or passing a variable populated asynchronously (e.g. from a queue, fetch, or loader) that resolved to `None` instead of a list.","commonSituations":"Async data pipelines where an `async def get_texts()` returns `None` on exhaustion; race conditions where a shared batch variable was reset to `None`; migrating sync `encode_batch` callers to the async API without fixing the `None` source.","solutions":["Pass a real list (use `[]` for an empty batch) instead of `None`.","Check `inputs is not None` before awaiting, or coalesce with `inputs or []`.","Fix the async producer that returned `None` instead of a list of sequences."],"exampleFix":"// before\nencodings = await tokenizer.async_encode_batch(texts)\n// after\nif texts is None:\n    texts = []\nencodings = await tokenizer.async_encode_batch(texts)","handlingStrategy":"validation","validationCode":"if inputs is None:\n    inputs = []\nencodings = await tokenizer.async_encode_batch(inputs)","typeGuard":"def is_ready_batch(value) -> bool:\n    return isinstance(value, list)","tryCatchPattern":"try:\n    encodings = await tokenizer.async_encode_batch(texts)\nexcept ValueError as e:\n    if \"can't be `None`\" in str(e):\n        encodings = []\n    else:\n        raise","preventionTips":["Make async producers return [] instead of None when a fetch yields nothing.","Coalesce with `inputs or []` at every await site.","Add an isinstance check right after awaiting data loaders."],"tags":["python","async","null-argument","tokenizers"],"backgroundTag":"null-argument","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}