{"record":{"id":"bd10dd3537ddf9e8","repo":"huggingface/tokenizers","slug":"async-encode-batch-fast-inputs-can-t-be-none","errorCode":null,"errorMessage":"async_encode_batch_fast: `inputs` can't be `None`","messagePattern":"async_encode_batch_fast: `inputs` can't be `None`","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"bindings/python/py_src/tokenizers/implementations/base_tokenizer.py","lineNumber":300,"sourceCode":"\n    async def async_encode_batch_fast(\n        self,\n        inputs: List[EncodeInput],\n        is_pretokenized: bool = False,\n        add_special_tokens: bool = True,\n    ) -> List[Encoding]:\n        \"\"\"Asynchronously encode a batch (no character offsets, faster).\n\n        Args:\n            inputs: A list of single or pair sequences to encode.\n            is_pretokenized: Whether inputs are already pre-tokenized.\n            add_special_tokens: Whether to add special tokens.\n\n        Returns:\n            A list of Encoding.\n        \"\"\"\n        if inputs is None:\n            raise ValueError(\"async_encode_batch_fast: `inputs` can't be `None`\")\n        return await self._tokenizer.async_encode_batch_fast(inputs, is_pretokenized, add_special_tokens)\n\n    def decode(self, ids: List[int], skip_special_tokens: Optional[bool] = True) -> str:\n        \"\"\"Decode the given list of ids to a string sequence\n\n        Args:\n            ids: List[unsigned int]:\n                A list of ids to be decoded\n\n            skip_special_tokens: (`optional`) boolean:\n                Whether to remove all the special tokens from the output string\n\n        Returns:\n            The decoded string\n        \"\"\"\n        if ids is None:\n            raise ValueError(\"None input is not valid. Should be a list of integers.\")\n","sourceCodeStart":282,"sourceCodeEnd":318,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/py_src/tokenizers/implementations/base_tokenizer.py#L282-L318","documentation":"`Tokenizer.async_encode_batch_fast` raises this `ValueError` when `inputs` is `None`. This is the faster async batch entry point; the Python wrapper still guards against `None` before invoking the Rust `async_encode_batch_fast`, rejecting null input with an explicit message.","triggerScenarios":"Calling `await tokenizer.async_encode_batch_fast(None)`, or forwarding a `None` result from an upstream loader/task into the fast batch API. Only `None` is rejected; `[]` is valid.","commonSituations":"High-throughput async inference services where the batching worker hands `None` to the tokenizer when no requests are pending; concurrency bugs where a batch slot was cleared to `None` just before encoding.","solutions":["Skip the call when the batch is `None`: encode only when `inputs is not None`.","Normalize to a list: `await tokenizer.async_encode_batch_fast(inputs or [])`.","Fix the batching worker so it yields `[]` rather than `None` for empty windows."],"exampleFix":"// before\nencodings = await tokenizer.async_encode_batch_fast(batch)\n// after\nencodings = await tokenizer.async_encode_batch_fast(batch) if batch is not None else []","handlingStrategy":"validation","validationCode":"if batch is None:\n    batch = []\nencodings = await tokenizer.async_encode_batch_fast(batch)","typeGuard":"def has_batch(value) -> bool:\n    return value is not None and isinstance(value, list)","tryCatchPattern":"try:\n    encodings = await tokenizer.async_encode_batch_fast(batch)\nexcept ValueError as e:\n    if \"can't be `None`\" in str(e):\n        encodings = []\n    else:\n        raise","preventionTips":["In batching workers, flush empty windows as [] rather than None.","Never reuse a batch variable slot cleared to None between iterations.","Type-hint the worker queue items as list, not Optional[list]."],"tags":["python","async","null-argument","tokenizers"],"backgroundTag":"null-argument","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}