huggingface/tokenizers · error · ValueError

async_encode_batch_fast: `inputs` can't be `None`

Error message

async_encode_batch_fast: `inputs` can't be `None`

What it means

`Tokenizer.async_encode_batch_fast` raises this `ValueError` when `inputs` is `None`. This is the faster async batch entry point; the Python wrapper still guards against `None` before invoking the Rust `async_encode_batch_fast`, rejecting null input with an explicit message.

Solutions

1. Skip the call when the batch is `None`: encode only when `inputs is not None`.
2. Normalize to a list: `await tokenizer.async_encode_batch_fast(inputs or [])`.
3. Fix the batching worker so it yields `[]` rather than `None` for empty windows.

Example fix

// before
encodings = await tokenizer.async_encode_batch_fast(batch)
// after
encodings = await tokenizer.async_encode_batch_fast(batch) if batch is not None else []

if batch is None:
    batch = []
encodings = await tokenizer.async_encode_batch_fast(batch)

def has_batch(value) -> bool:
    return value is not None and isinstance(value, list)

try:
    encodings = await tokenizer.async_encode_batch_fast(batch)
except ValueError as e:
    if "can't be `None`" in str(e):
        encodings = []
    else:
        raise

When it happens

Trigger: Calling `await tokenizer.async_encode_batch_fast(None)`, or forwarding a `None` result from an upstream loader/task into the fast batch API. Only `None` is rejected; `[]` is valid.

Common situations: High-throughput async inference services where the batching worker hands `None` to the tokenizer when no requests are pending; concurrency bugs where a batch slot was cleared to `None` just before encoding.

Related errors

• async_encode_batch: `inputs` can't be `None`
• async_decode_batch: `sequences` can't be `None`
• encode: `sequence` can't be `None`
• encode_batch: `inputs` can't be `None`
• None input is not valid. Should be a list of integers.

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/bd10dd3537ddf9e8. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/implementations/base_tokenizer.py:300

    async def async_encode_batch_fast(
        self,
        inputs: List[EncodeInput],
        is_pretokenized: bool = False,
        add_special_tokens: bool = True,
    ) -> List[Encoding]:
        """Asynchronously encode a batch (no character offsets, faster).

        Args:
            inputs: A list of single or pair sequences to encode.
            is_pretokenized: Whether inputs are already pre-tokenized.
            add_special_tokens: Whether to add special tokens.

        Returns:
            A list of Encoding.
        """
        if inputs is None:
            raise ValueError("async_encode_batch_fast: `inputs` can't be `None`")
        return await self._tokenizer.async_encode_batch_fast(inputs, is_pretokenized, add_special_tokens)

    def decode(self, ids: List[int], skip_special_tokens: Optional[bool] = True) -> str:
        """Decode the given list of ids to a string sequence

        Args:
            ids: List[unsigned int]:
                A list of ids to be decoded

            skip_special_tokens: (`optional`) boolean:
                Whether to remove all the special tokens from the output string

        Returns:
            The decoded string
        """
        if ids is None:
            raise ValueError("None input is not valid. Should be a list of integers.")

View on GitHub (pinned to 6cfd9d385c)