{"record":{"id":"8a1d010ef72ab2d7","repo":"huggingface/tokenizers","slug":"none-input-is-not-valid-should-be-list-of-list-of","errorCode":null,"errorMessage":"None input is not valid. Should be list of list of integers.","messagePattern":"None input is not valid\\. Should be list of list of integers\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"bindings/python/py_src/tokenizers/implementations/base_tokenizer.py","lineNumber":335,"sourceCode":"            raise ValueError(\"None input is not valid. Should be a list of integers.\")\n\n        return self._tokenizer.decode(ids, skip_special_tokens=skip_special_tokens)\n\n    def decode_batch(self, sequences: List[List[int]], skip_special_tokens: Optional[bool] = True) -> str:\n        \"\"\"Decode the list of sequences to a list of string sequences\n\n        Args:\n            sequences: List[List[unsigned int]]:\n                A list of sequence of ids to be decoded\n\n            skip_special_tokens: (`optional`) boolean:\n                Whether to remove all the special tokens from the output strings\n\n        Returns:\n            A list of decoded strings\n        \"\"\"\n        if sequences is None:\n            raise ValueError(\"None input is not valid. Should be list of list of integers.\")\n\n        return self._tokenizer.decode_batch(sequences, skip_special_tokens=skip_special_tokens)\n\n    async def async_decode_batch(\n        self,\n        sequences: List[List[int]],\n        skip_special_tokens: bool = True,\n    ) -> List[str]:\n        \"\"\"Asynchronously decode a batch of sequences.\n\n        Args:\n            sequences: A list of sequences of ids to decode.\n            skip_special_tokens: Whether to remove special tokens from output.\n\n        Returns:\n            A list of decoded strings.\n        \"\"\"\n        if sequences is None:","sourceCodeStart":317,"sourceCodeEnd":353,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/py_src/tokenizers/implementations/base_tokenizer.py#L317-L353","documentation":"`Tokenizer.decode_batch` raises this `ValueError` when `sequences` is `None`. The method expects a list of lists of integer ids; the wrapper checks for `None` before delegating to the Rust implementation so the failure message states the expected shape.","triggerScenarios":"Calling `tokenizer.decode_batch(None)`, or passing a `None` produced by an upstream aggregation step (e.g. `[enc.ids for enc in encodings]` where `encodings` was `None`).","commonSituations":"Batch post-processing in inference servers where the encoder stage failed silently and returned `None`; refactors that swapped the argument order or passed a single id list instead of a list of lists.","solutions":["Pass a list of id lists: `tokenizer.decode_batch([[101, 102], [103, 104]])`.","Coalesce to an empty batch: `tokenizer.decode_batch(sequences or [])`.","Fix the producer that returned `None` instead of `List[List[int]]`."],"exampleFix":"// before\ntexts = tokenizer.decode_batch(seqs)\n// after\ntexts = tokenizer.decode_batch(seqs) if seqs is not None else []","handlingStrategy":"validation","validationCode":"if sequences is None:\n    sequences = []\ntexts = tokenizer.decode_batch(sequences)","typeGuard":"def is_seq_of_ids(value) -> bool:\n    return isinstance(value, list) and all(isinstance(s, list) for s in value)","tryCatchPattern":"try:\n    texts = tokenizer.decode_batch(seqs)\nexcept ValueError as e:\n    if \"None input is not valid\" in str(e):\n        texts = []\n    else:\n        raise","preventionTips":["Build batches with list comprehensions over encodings so the result is always a list.","Wrap encode/decode pairs in helpers that assert list-of-list shapes.","Fix producers to return [] instead of None on empty input."],"tags":["python","null-argument","tokenizers"],"backgroundTag":"null-argument","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}