huggingface/tokenizers · error · ValueError

None input is not valid. Should be a list of integers.

Error message

None input is not valid. Should be a list of integers.

What it means

`Tokenizer.decode` raises this `ValueError` when the `ids` argument is `None`. Decoding requires a list of integer token ids; the wrapper rejects `None` up front with a message describing the expected type (`a list of integers`) instead of letting the Rust layer fail obscurely.

Solutions

1. Pass an actual list of ints: `tokenizer.decode([101, 2023, 102])`.
2. Guard before decoding: `if ids is not None: text = tokenizer.decode(ids)`.
3. Extract the ids correctly, e.g. `tokenizer.decode(encoding.ids)` on the `Encoding` object, not on a `None` value.

Example fix

// before
text = tokenizer.decode(ids)
// after
text = tokenizer.decode(ids) if ids is not None else ""

if ids is not None:
    assert all(isinstance(i, int) for i in ids), "decode expects list of ints"
text = tokenizer.decode(ids or [])

def is_id_list(value) -> bool:
    return isinstance(value, list) and all(isinstance(i, int) for i in value)

try:
    text = tokenizer.decode(ids)
except ValueError as e:
    if "None input is not valid" in str(e):
        text = ""
    else:
        raise

When it happens

Trigger: Calling `tokenizer.decode(None)`, or passing a variable that holds `None` because id extraction failed (e.g. `encoding["ids"]` on a missing key, or a lookup returning `None`).

Common situations: Round-tripping code where `encode` returned an unexpected structure and `ids` stayed `None`; model-output post-processing where an optional tensor was converted to `None`; notebooks copying `tokenizer.decode(output)` where `output` is a dict without ids.

Related errors

• encode: `sequence` can't be `None`
• encode_batch: `inputs` can't be `None`
• async_encode_batch: `inputs` can't be `None`
• async_encode_batch_fast: `inputs` can't be `None`
• None input is not valid. Should be list of list of integers.

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/1fc1977b16bcaa7b. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/implementations/base_tokenizer.py:317

        if inputs is None:
            raise ValueError("async_encode_batch_fast: `inputs` can't be `None`")
        return await self._tokenizer.async_encode_batch_fast(inputs, is_pretokenized, add_special_tokens)

    def decode(self, ids: List[int], skip_special_tokens: Optional[bool] = True) -> str:
        """Decode the given list of ids to a string sequence

        Args:
            ids: List[unsigned int]:
                A list of ids to be decoded

            skip_special_tokens: (`optional`) boolean:
                Whether to remove all the special tokens from the output string

        Returns:
            The decoded string
        """
        if ids is None:
            raise ValueError("None input is not valid. Should be a list of integers.")

        return self._tokenizer.decode(ids, skip_special_tokens=skip_special_tokens)

    def decode_batch(self, sequences: List[List[int]], skip_special_tokens: Optional[bool] = True) -> str:
        """Decode the list of sequences to a list of string sequences

        Args:
            sequences: List[List[unsigned int]]:
                A list of sequence of ids to be decoded

            skip_special_tokens: (`optional`) boolean:
                Whether to remove all the special tokens from the output strings

        Returns:
            A list of decoded strings
        """
        if sequences is None:
            raise ValueError("None input is not valid. Should be list of list of integers.")

View on GitHub (pinned to 6cfd9d385c)