{"record":{"id":"dbd8f19715f1ded1","repo":"karpathy/nanochat","slug":"invalid-input-type-type-text","errorCode":null,"errorMessage":"Invalid input type: {type(text)}","messagePattern":"Invalid input type: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"nanochat/tokenizer.py","lineNumber":119,"sourceCode":"        if append is not None:\n            append_id = append if isinstance(append, int) else self.encode_special(append)\n\n        if isinstance(text, str):\n            ids = self.enc.encode_ordinary(text)\n            if prepend is not None:\n                ids.insert(0, prepend_id) # TODO: slightly inefficient here? :( hmm\n            if append is not None:\n                ids.append(append_id)\n        elif isinstance(text, list):\n            ids = self.enc.encode_ordinary_batch(text, num_threads=num_threads)\n            if prepend is not None:\n                for ids_row in ids:\n                    ids_row.insert(0, prepend_id) # TODO: same\n            if append is not None:\n                for ids_row in ids:\n                    ids_row.append(append_id)\n        else:\n            raise ValueError(f\"Invalid input type: {type(text)}\")\n\n        return ids\n\n    def __call__(self, *args, **kwargs):\n        return self.encode(*args, **kwargs)\n\n    def decode(self, ids):\n        return self.enc.decode(ids)\n\n    def decode_single_token_bytes(self, token_id):\n        return self.enc.decode_single_token_bytes(token_id)\n\n    def save(self, tokenizer_dir):\n        # save the encoding object to disk\n        os.makedirs(tokenizer_dir, exist_ok=True)\n        pickle_path = os.path.join(tokenizer_dir, \"tokenizer.pkl\")\n        with open(pickle_path, \"wb\") as f:\n            pickle.dump(self.enc, f)","sourceCodeStart":101,"sourceCodeEnd":137,"githubUrl":"https://github.com/karpathy/nanochat/blob/92d63d4e8bb4df75c3b71618f31ddde2378b2bcd/nanochat/tokenizer.py#L101-L137","documentation":"Tokenizer.encode accepts exactly two input shapes: a single string (returns list[int]) or a list of strings (returns list[list[int]] via encode_ordinary_batch). Anything else — int, None, bytes, a numpy array, a torch.Tensor, a nested list — reaches the else and raises ValueError with the offending type.","triggerScenarios":"Calling tokenizer.encode(123), tokenizer.encode(None), tokenizer.encode([1,2,3]) (a list of ints, not strings), or passing a torch tensor/numpy array of text instead of Python strings.","commonSituations":"Feeding already-tokenized ids back into encode; passing a numpy array of strings from a data pipeline; passing None from a missing dataset field; mixing up encode and decode directions.","solutions":["Convert input to str or list[str] before calling encode: `tokenizer.encode(text.item())` for 0-d numpy, `.tolist()` for tensors of ids is a decode-side operation.","If input is already token ids, use `tokenizer.decode(ids)` instead of encode.","For batch mode, ensure every element is a str: `all(isinstance(t, str) for t in texts)`."],"exampleFix":"# before\nids = tokenizer.encode(example[\"tokens\"])  # list[int], not list[str]\n\n# after\ntext = tokenizer.decode(example[\"tokens\"])\nids = tokenizer.encode(text)","handlingStrategy":"type-guard","validationCode":"if not isinstance(text, (str, list)) or (isinstance(text, list) and not all(isinstance(t, str) for t in text)):\n    raise TypeError(f\"encode expects str or list[str], got {type(text).__name__}\")","typeGuard":"def is_valid_encode_input(text) -> bool:\n    if isinstance(text, str):\n        return True\n    return isinstance(text, list) and all(isinstance(t, str) for t in text)","tryCatchPattern":"try:\n    ids = tokenizer.encode(text)\nexcept ValueError:\n    text = str(text) if not isinstance(text, list) else [str(t) for t in text]\n    ids = tokenizer.encode(text)","preventionTips":["Coerce numpy/tensor scalars to Python str before encoding.","Remember ids go to decode(), strings go to encode() — never mix directions.","For batch inputs, verify every element is str."],"tags":["nanochat","tokenizer","type-error","validation"],"backgroundTag":null,"analyzedSha":"92d63d4e8bb4df75c3b71618f31ddde2378b2bcd","analyzedAt":"2026-08-15T03:11:54.371Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}