huggingface/tokenizers · error · ValueError

encode: `sequence` can't be `None`

Error message

encode: `sequence` can't be `None`

What it means

Tokenizer.encode() in bindings/python/py_src/tokenizers/implementations/base_tokenizer.py raises ValueError when the `sequence` argument is None before delegating to the Rust tokenizer. The underlying API requires a text sequence (optionally with a pair), and None is not a valid input, so the Python wrapper validates it explicitly.

Solutions

1. Check the input for None before calling encode and skip/replace it (e.g. with the empty string '')
2. Fix the upstream data source so the field is never None (default to '' when missing)
3. If you intended to encode a pair, pass it via the `pair` argument, not as a None first argument

Example fix

// before
tokenizer.encode(row.get("text"))  # ValueError if text is None
// after
text = row.get("text") or ""
tokenizer.encode(text)

def safe_encode(tokenizer, sequence, **kwargs):
    if sequence is None:
        raise ValueError("Refusing to encode: sequence is None")
    return tokenizer.encode(sequence, **kwargs)

def is_valid_sequence(sequence) -> bool:
    return sequence is not None and isinstance(sequence, str)

try:
    encoding = tokenizer.encode(sequence)
except ValueError as e:
    if "can't be `None`" in str(e):
        logging.warning("Skipping None input to encode")
        encoding = None
    else:
        raise

When it happens

Trigger: Calling tokenizer.encode(None) — directly or via a pipeline where the input variable was never populated — with signature encode(sequence, pair=None, is_pretokenized=False, add_special_tokens=True).

Common situations: Batch/ETL pipelines where a text field is missing and becomes None; results of a previous tokenization step (e.g. .text) that is None; passing the output of a failed lookup straight into encode without checking.

Related errors

• encode_batch: `inputs` can't be `None`
• async_encode_batch: `inputs` can't be `None`
• async_encode_batch_fast: `inputs` can't be `None`
• None input is not valid. Should be a list of integers.
• None input is not valid. Should be list of list of integers.

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/9a07d7e95a521d8b. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/implementations/base_tokenizer.py:221

            sequence: InputSequence:
                The sequence we want to encode. This sequence can be either raw text or
                pre-tokenized, according to the `is_pretokenized` argument:

                - If `is_pretokenized=False`: `InputSequence` is expected to be `str`
                - If `is_pretokenized=True`: `InputSequence` is expected to be
                    `Union[List[str], Tuple[str]]`

            is_pretokenized: bool:
                Whether the input is already pre-tokenized.

            add_special_tokens: bool:
                Whether to add the special tokens while encoding.

        Returns:
            An Encoding
        """
        if sequence is None:
            raise ValueError("encode: `sequence` can't be `None`")

        return self._tokenizer.encode(sequence, pair, is_pretokenized, add_special_tokens)

    def encode_batch(
        self,
        inputs: List[EncodeInput],
        is_pretokenized: bool = False,
        add_special_tokens: bool = True,
    ) -> List[Encoding]:
        """Encode the given inputs. This method accept both raw text sequences as well as already
        pre-tokenized sequences.

        Args:
            inputs: List[EncodeInput]:
                A list of single sequences or pair sequences to encode. Each `EncodeInput` is
                expected to be of the following form:
                    `Union[InputSequence, Tuple[InputSequence, InputSequence]]`

View on GitHub (pinned to 6cfd9d385c)