{"record":{"id":"1fc1977b16bcaa7b","repo":"huggingface/tokenizers","slug":"none-input-is-not-valid-should-be-a-list-of-integ","errorCode":null,"errorMessage":"None input is not valid. Should be a list of integers.","messagePattern":"None input is not valid\\. Should be a list of integers\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"bindings/python/py_src/tokenizers/implementations/base_tokenizer.py","lineNumber":317,"sourceCode":"        if inputs is None:\n            raise ValueError(\"async_encode_batch_fast: `inputs` can't be `None`\")\n        return await self._tokenizer.async_encode_batch_fast(inputs, is_pretokenized, add_special_tokens)\n\n    def decode(self, ids: List[int], skip_special_tokens: Optional[bool] = True) -> str:\n        \"\"\"Decode the given list of ids to a string sequence\n\n        Args:\n            ids: List[unsigned int]:\n                A list of ids to be decoded\n\n            skip_special_tokens: (`optional`) boolean:\n                Whether to remove all the special tokens from the output string\n\n        Returns:\n            The decoded string\n        \"\"\"\n        if ids is None:\n            raise ValueError(\"None input is not valid. Should be a list of integers.\")\n\n        return self._tokenizer.decode(ids, skip_special_tokens=skip_special_tokens)\n\n    def decode_batch(self, sequences: List[List[int]], skip_special_tokens: Optional[bool] = True) -> str:\n        \"\"\"Decode the list of sequences to a list of string sequences\n\n        Args:\n            sequences: List[List[unsigned int]]:\n                A list of sequence of ids to be decoded\n\n            skip_special_tokens: (`optional`) boolean:\n                Whether to remove all the special tokens from the output strings\n\n        Returns:\n            A list of decoded strings\n        \"\"\"\n        if sequences is None:\n            raise ValueError(\"None input is not valid. Should be list of list of integers.\")","sourceCodeStart":299,"sourceCodeEnd":335,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/py_src/tokenizers/implementations/base_tokenizer.py#L299-L335","documentation":"`Tokenizer.decode` raises this `ValueError` when the `ids` argument is `None`. Decoding requires a list of integer token ids; the wrapper rejects `None` up front with a message describing the expected type (`a list of integers`) instead of letting the Rust layer fail obscurely.","triggerScenarios":"Calling `tokenizer.decode(None)`, or passing a variable that holds `None` because id extraction failed (e.g. `encoding[\"ids\"]` on a missing key, or a lookup returning `None`).","commonSituations":"Round-tripping code where `encode` returned an unexpected structure and `ids` stayed `None`; model-output post-processing where an optional tensor was converted to `None`; notebooks copying `tokenizer.decode(output)` where `output` is a dict without ids.","solutions":["Pass an actual list of ints: `tokenizer.decode([101, 2023, 102])`.","Guard before decoding: `if ids is not None: text = tokenizer.decode(ids)`.","Extract the ids correctly, e.g. `tokenizer.decode(encoding.ids)` on the `Encoding` object, not on a `None` value."],"exampleFix":"// before\ntext = tokenizer.decode(ids)\n// after\ntext = tokenizer.decode(ids) if ids is not None else \"\"","handlingStrategy":"type-guard","validationCode":"if ids is not None:\n    assert all(isinstance(i, int) for i in ids), \"decode expects list of ints\"\ntext = tokenizer.decode(ids or [])","typeGuard":"def is_id_list(value) -> bool:\n    return isinstance(value, list) and all(isinstance(i, int) for i in value)","tryCatchPattern":"try:\n    text = tokenizer.decode(ids)\nexcept ValueError as e:\n    if \"None input is not valid\" in str(e):\n        text = \"\"\n    else:\n        raise","preventionTips":["Use `encoding.ids` from the Encoding object rather than dict indexing that can yield None.","Keep ids extraction and decoding in one typed helper function.","Check the encode step's return type when refactoring encode->decode pipelines."],"tags":["python","null-argument","tokenizers"],"backgroundTag":"null-argument","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}