huggingface/tokenizers · error · ValueError

async_decode_batch: `sequences` can't be `None`

Error message

async_decode_batch: `sequences` can't be `None`

What it means

`Tokenizer.async_decode_batch` raises this `ValueError` when `sequences` is `None`. The async wrapper validates that the batch of id sequences is a list before awaiting the Rust implementation, giving a clear message naming the offending parameter.

Solutions

1. Pass a list of id lists, or `[]` for an empty batch.
2. Guard the await: `if seqs is not None: texts = await tokenizer.async_decode_batch(seqs)`.
3. Fix the upstream async stage to return `[]` instead of `None`.

Example fix

// before
texts = await tokenizer.async_decode_batch(seqs)
// after
texts = await tokenizer.async_decode_batch(seqs or [])

if seqs is None:
    seqs = []
texts = await tokenizer.async_decode_batch(seqs)

def is_batch_of_ids(value) -> bool:
    return isinstance(value, list) and all(isinstance(s, list) for s in value)

try:
    texts = await tokenizer.async_decode_batch(seqs)
except ValueError as e:
    if "can't be `None`" in str(e):
        texts = []
    else:
        raise

When it happens

Trigger: Calling `await tokenizer.async_decode_batch(None)`, or awaiting with a variable that a prior async stage set to `None` (failed encode batch, cancelled aggregation).

Common situations: Async serving stacks where decode is chained after async encode and an earlier failure left the batch as `None`; pipeline frameworks that represent 'no data' as `None` instead of `[]`.

Related errors

• async_encode_batch: `inputs` can't be `None`
• async_encode_batch_fast: `inputs` can't be `None`
• encode: `sequence` can't be `None`
• encode_batch: `inputs` can't be `None`
• None input is not valid. Should be a list of integers.

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/94be830c467ded2f. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/implementations/base_tokenizer.py:354

        return self._tokenizer.decode_batch(sequences, skip_special_tokens=skip_special_tokens)

    async def async_decode_batch(
        self,
        sequences: List[List[int]],
        skip_special_tokens: bool = True,
    ) -> List[str]:
        """Asynchronously decode a batch of sequences.

        Args:
            sequences: A list of sequences of ids to decode.
            skip_special_tokens: Whether to remove special tokens from output.

        Returns:
            A list of decoded strings.
        """
        if sequences is None:
            raise ValueError("async_decode_batch: `sequences` can't be `None`")
        return await self._tokenizer.async_decode_batch(sequences, skip_special_tokens)

    def token_to_id(self, token: str) -> Optional[int]:
        """Convert the given token to its corresponding id

        Args:
            token: str:
                The token to convert

        Returns:
            The corresponding id if it exists, None otherwise
        """
        return self._tokenizer.token_to_id(token)

    def id_to_token(self, id: int) -> Optional[str]:
        """Convert the given token id to its corresponding string

        Args:

View on GitHub (pinned to 6cfd9d385c)