What causes "None input is not valid. Should be list of list of integers." in huggingface/tokenizers?

Pass a list of id lists: `tokenizer.decode_batch([[101, 102], [103, 104]])`.

huggingface/tokenizers · error · ValueError

None input is not valid. Should be list of list of integers.

Error message

None input is not valid. Should be list of list of integers.

What it means

`Tokenizer.decode_batch` raises this `ValueError` when `sequences` is `None`. The method expects a list of lists of integer ids; the wrapper checks for `None` before delegating to the Rust implementation so the failure message states the expected shape.

Solutions

Pass a list of id lists: `tokenizer.decode_batch([[101, 102], [103, 104]])`.
Coalesce to an empty batch: `tokenizer.decode_batch(sequences or [])`.
Fix the producer that returned `None` instead of `List[List[int]]`.

Example fix

// before
texts = tokenizer.decode_batch(seqs)
// after
texts = tokenizer.decode_batch(seqs) if seqs is not None else []

Defensive patterns

Strategy: validation

Validate before calling

if sequences is None:
    sequences = []
texts = tokenizer.decode_batch(sequences)

Type guard

def is_seq_of_ids(value) -> bool:
    return isinstance(value, list) and all(isinstance(s, list) for s in value)

Try / catch

try:
    texts = tokenizer.decode_batch(seqs)
except ValueError as e:
    if "None input is not valid" in str(e):
        texts = []
    else:
        raise

Prevention

Build batches with list comprehensions over encodings so the result is always a list.
Wrap encode/decode pairs in helpers that assert list-of-list shapes.
Fix producers to return [] instead of None on empty input.

When it happens

Trigger: Calling `tokenizer.decode_batch(None)`, or passing a `None` produced by an upstream aggregation step (e.g. `[enc.ids for enc in encodings]` where `encodings` was `None`).

Common situations: Batch post-processing in inference servers where the encoder stage failed silently and returned `None`; refactors that swapped the argument order or passed a single id list instead of a list of lists.

Related errors

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/8a1d010ef72ab2d7. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/implementations/base_tokenizer.py:335

            raise ValueError("None input is not valid. Should be a list of integers.")

        return self._tokenizer.decode(ids, skip_special_tokens=skip_special_tokens)

    def decode_batch(self, sequences: List[List[int]], skip_special_tokens: Optional[bool] = True) -> str:
        """Decode the list of sequences to a list of string sequences

        Args:
            sequences: List[List[unsigned int]]:
                A list of sequence of ids to be decoded

            skip_special_tokens: (`optional`) boolean:
                Whether to remove all the special tokens from the output strings

        Returns:
            A list of decoded strings
        """
        if sequences is None:
            raise ValueError("None input is not valid. Should be list of list of integers.")

        return self._tokenizer.decode_batch(sequences, skip_special_tokens=skip_special_tokens)

    async def async_decode_batch(
        self,
        sequences: List[List[int]],
        skip_special_tokens: bool = True,
    ) -> List[str]:
        """Asynchronously decode a batch of sequences.

        Args:
            sequences: A list of sequences of ids to decode.
            skip_special_tokens: Whether to remove special tokens from output.

        Returns:
            A list of decoded strings.
        """
        if sequences is None:

View on GitHub (pinned to 6cfd9d385c)