{"record":{"id":"9a07d7e95a521d8b","repo":"huggingface/tokenizers","slug":"encode-sequence-can-t-be-none","errorCode":null,"errorMessage":"encode: `sequence` can't be `None`","messagePattern":"encode: `sequence` can't be `None`","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"bindings/python/py_src/tokenizers/implementations/base_tokenizer.py","lineNumber":221,"sourceCode":"            sequence: InputSequence:\n                The sequence we want to encode. This sequence can be either raw text or\n                pre-tokenized, according to the `is_pretokenized` argument:\n\n                - If `is_pretokenized=False`: `InputSequence` is expected to be `str`\n                - If `is_pretokenized=True`: `InputSequence` is expected to be\n                    `Union[List[str], Tuple[str]]`\n\n            is_pretokenized: bool:\n                Whether the input is already pre-tokenized.\n\n            add_special_tokens: bool:\n                Whether to add the special tokens while encoding.\n\n        Returns:\n            An Encoding\n        \"\"\"\n        if sequence is None:\n            raise ValueError(\"encode: `sequence` can't be `None`\")\n\n        return self._tokenizer.encode(sequence, pair, is_pretokenized, add_special_tokens)\n\n    def encode_batch(\n        self,\n        inputs: List[EncodeInput],\n        is_pretokenized: bool = False,\n        add_special_tokens: bool = True,\n    ) -> List[Encoding]:\n        \"\"\"Encode the given inputs. This method accept both raw text sequences as well as already\n        pre-tokenized sequences.\n\n        Args:\n            inputs: List[EncodeInput]:\n                A list of single sequences or pair sequences to encode. Each `EncodeInput` is\n                expected to be of the following form:\n                    `Union[InputSequence, Tuple[InputSequence, InputSequence]]`\n","sourceCodeStart":203,"sourceCodeEnd":239,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/py_src/tokenizers/implementations/base_tokenizer.py#L203-L239","documentation":"Tokenizer.encode() in bindings/python/py_src/tokenizers/implementations/base_tokenizer.py raises ValueError when the `sequence` argument is None before delegating to the Rust tokenizer. The underlying API requires a text sequence (optionally with a pair), and None is not a valid input, so the Python wrapper validates it explicitly.","triggerScenarios":"Calling tokenizer.encode(None) — directly or via a pipeline where the input variable was never populated — with signature encode(sequence, pair=None, is_pretokenized=False, add_special_tokens=True).","commonSituations":"Batch/ETL pipelines where a text field is missing and becomes None; results of a previous tokenization step (e.g. .text) that is None; passing the output of a failed lookup straight into encode without checking.","solutions":["Check the input for None before calling encode and skip/replace it (e.g. with the empty string '')","Fix the upstream data source so the field is never None (default to '' when missing)","If you intended to encode a pair, pass it via the `pair` argument, not as a None first argument"],"exampleFix":"// before\ntokenizer.encode(row.get(\"text\"))  # ValueError if text is None\n// after\ntext = row.get(\"text\") or \"\"\ntokenizer.encode(text)","handlingStrategy":"validation","validationCode":"def safe_encode(tokenizer, sequence, **kwargs):\n    if sequence is None:\n        raise ValueError(\"Refusing to encode: sequence is None\")\n    return tokenizer.encode(sequence, **kwargs)","typeGuard":"def is_valid_sequence(sequence) -> bool:\n    return sequence is not None and isinstance(sequence, str)","tryCatchPattern":"try:\n    encoding = tokenizer.encode(sequence)\nexcept ValueError as e:\n    if \"can't be `None`\" in str(e):\n        logging.warning(\"Skipping None input to encode\")\n        encoding = None\n    else:\n        raise","preventionTips":["Coalesce None inputs to '' (or skip) before calling encode","Add data validation upstream so text fields are never None","Use tokenizer.encode_batch with pre-filtered inputs to avoid partial failures","Write unit tests covering missing/null text fields in your pipeline"],"tags":["python","null-argument","tokenizers","validation"],"backgroundTag":"null-argument","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}