What causes "async_encode_batch: `inputs` can't be `None`" in huggingface/tokenizers?

Pass a real list (use `[]` for an empty batch) instead of `None`.

huggingface/tokenizers · error · ValueError

async_encode_batch: `inputs` can't be `None`

Error message

async_encode_batch: `inputs` can't be `None`

What it means

`Tokenizer.async_encode_batch` raises this `ValueError` when `inputs` is `None`. As with the sync variant, the Python layer validates the argument before awaiting the Rust implementation exposed via pyo3, so a null batch is rejected with a clear message rather than a binding-level failure.

Solutions

Pass a real list (use `[]` for an empty batch) instead of `None`.
Check `inputs is not None` before awaiting, or coalesce with `inputs or []`.
Fix the async producer that returned `None` instead of a list of sequences.

Example fix

// before
encodings = await tokenizer.async_encode_batch(texts)
// after
if texts is None:
    texts = []
encodings = await tokenizer.async_encode_batch(texts)

Defensive patterns

Strategy: validation

Validate before calling

if inputs is None:
    inputs = []
encodings = await tokenizer.async_encode_batch(inputs)

Type guard

def is_ready_batch(value) -> bool:
    return isinstance(value, list)

Try / catch

try:
    encodings = await tokenizer.async_encode_batch(texts)
except ValueError as e:
    if "can't be `None`" in str(e):
        encodings = []
    else:
        raise

Prevention

Make async producers return [] instead of None when a fetch yields nothing.
Coalesce with `inputs or []` at every await site.
Add an isinstance check right after awaiting data loaders.

When it happens

Trigger: Awaiting `await tokenizer.async_encode_batch(None)`, or passing a variable populated asynchronously (e.g. from a queue, fetch, or loader) that resolved to `None` instead of a list.

Common situations: Async data pipelines where an `async def get_texts()` returns `None` on exhaustion; race conditions where a shared batch variable was reset to `None`; migrating sync `encode_batch` callers to the async API without fixing the `None` source.

Related errors

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/af7b06b6a2e608e5. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/implementations/base_tokenizer.py:279

    async def async_encode_batch(
        self,
        inputs: List[EncodeInput],
        is_pretokenized: bool = False,
        add_special_tokens: bool = True,
    ) -> List[Encoding]:
        """Asynchronously encode a batch (tracks character offsets).

        Args:
            inputs: A list of single or pair sequences to encode.
            is_pretokenized: Whether inputs are already pre-tokenized.
            add_special_tokens: Whether to add special tokens.

        Returns:
            A list of Encoding.
        """
        if inputs is None:
            raise ValueError("async_encode_batch: `inputs` can't be `None`")
        # Exposed by the Rust bindings via pyo3_async_runtimes::tokio::future_into_py
        return await self._tokenizer.async_encode_batch(inputs, is_pretokenized, add_special_tokens)

    async def async_encode_batch_fast(
        self,
        inputs: List[EncodeInput],
        is_pretokenized: bool = False,
        add_special_tokens: bool = True,
    ) -> List[Encoding]:
        """Asynchronously encode a batch (no character offsets, faster).

        Args:
            inputs: A list of single or pair sequences to encode.
            is_pretokenized: Whether inputs are already pre-tokenized.
            add_special_tokens: Whether to add special tokens.

        Returns:
            A list of Encoding.

View on GitHub (pinned to 6cfd9d385c)