huggingface/tokenizers · error · ValueError

encode_batch: `inputs` can't be `None`

Error message

encode_batch: `inputs` can't be `None`

What it means

`Tokenizer.encode_batch` raises this `ValueError` when the `inputs` parameter is `None`. The Python wrapper performs an explicit null check before delegating to the Rust tokenizer, because `None` would otherwise surface as an opaque panic or type error inside the bindings. Passing `None` instead of an empty list or a list of sequences is almost always an upstream bug where a batch variable was never populated.

Solutions

1. Ensure the argument is a list: pass `[]` if there is truly nothing to encode, or the list of input strings/pre-tokenized sequences.
2. Guard before calling: `if texts is not None: encodings = tok.encode_batch(texts)`.
3. Fix the upstream function that produced `None` instead of a list (e.g. return `[]` on empty input).

Example fix

// before
encodings = tokenizer.encode_batch(batch)
// after
encodings = tokenizer.encode_batch(batch or [])

if not isinstance(inputs, list):
    raise TypeError(f"encode_batch expects a list, got {type(inputs).__name__}")
encodings = tokenizer.encode_batch(inputs or [])

def is_batch(value) -> bool:
    return isinstance(value, list)

try:
    encodings = tokenizer.encode_batch(texts)
except ValueError as e:
    if "can't be `None`" in str(e):
        encodings = []
    else:
        raise

When it happens

Trigger: Calling `tokenizer.encode_batch(None)` directly, or calling it with a variable that a caller/factory returned as `None` (e.g. `encode_batch(batch)` where `batch` was never initialized). Only `None` triggers it; an empty list `[]` is accepted and returns `[]`.

Common situations: Batching loops where `texts = get_batch()` can return `None` at the end of a dataset; config-driven pipelines where a missing dataset key defaults to `None`; passing the result of a failed file read (`open(...).read()` inside try, variable stays `None`).

Related errors

• encode: `sequence` can't be `None`
• async_encode_batch: `inputs` can't be `None`
• async_encode_batch_fast: `inputs` can't be `None`
• None input is not valid. Should be a list of integers.
• None input is not valid. Should be list of list of integers.

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/37d429c31660acd8. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/implementations/base_tokenizer.py:258

                Each `InputSequence` can either be raw text or pre-tokenized,
                according to the `is_pretokenized` argument:

                - If `is_pretokenized=False`: `InputSequence` is expected to be `str`
                - If `is_pretokenized=True`: `InputSequence` is expected to be
                    `Union[List[str], Tuple[str]]`

            is_pretokenized: bool:
                Whether the input is already pre-tokenized.

            add_special_tokens: bool:
                Whether to add the special tokens while encoding.

        Returns:
            A list of Encoding
        """

        if inputs is None:
            raise ValueError("encode_batch: `inputs` can't be `None`")

        return self._tokenizer.encode_batch(inputs, is_pretokenized, add_special_tokens)

    async def async_encode_batch(
        self,
        inputs: List[EncodeInput],
        is_pretokenized: bool = False,
        add_special_tokens: bool = True,
    ) -> List[Encoding]:
        """Asynchronously encode a batch (tracks character offsets).

        Args:
            inputs: A list of single or pair sequences to encode.
            is_pretokenized: Whether inputs are already pre-tokenized.
            add_special_tokens: Whether to add special tokens.

        Returns:
            A list of Encoding.

View on GitHub (pinned to 6cfd9d385c)