{"record":{"id":"5f99ff82e6c2fd7b","repo":"karpathy/autoresearch","slug":"invalid-input-type-type-text","errorCode":null,"errorMessage":"Invalid input type: {type(text)}","messagePattern":"Invalid input type: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"prepare.py","lineNumber":241,"sourceCode":"        return self.enc.n_vocab\n\n    def get_bos_token_id(self):\n        return self.bos_token_id\n\n    def encode(self, text, prepend=None, num_threads=8):\n        if prepend is not None:\n            prepend_id = prepend if isinstance(prepend, int) else self.enc.encode_single_token(prepend)\n        if isinstance(text, str):\n            ids = self.enc.encode_ordinary(text)\n            if prepend is not None:\n                ids.insert(0, prepend_id)\n        elif isinstance(text, list):\n            ids = self.enc.encode_ordinary_batch(text, num_threads=num_threads)\n            if prepend is not None:\n                for row in ids:\n                    row.insert(0, prepend_id)\n        else:\n            raise ValueError(f\"Invalid input type: {type(text)}\")\n        return ids\n\n    def decode(self, ids):\n        return self.enc.decode(ids)\n\n\ndef get_token_bytes(device=\"cpu\"):\n    path = os.path.join(TOKENIZER_DIR, \"token_bytes.pt\")\n    with open(path, \"rb\") as f:\n        return torch.load(f, map_location=device)\n\n\ndef _document_batches(split, tokenizer_batch_size=128):\n    \"\"\"Infinite iterator over document batches from parquet files.\"\"\"\n    parquet_paths = list_parquet_files()\n    assert len(parquet_paths) > 0, \"No parquet files found. Run prepare.py first.\"\n    val_path = os.path.join(DATA_DIR, VAL_FILENAME)\n    if split == \"train\":","sourceCodeStart":223,"sourceCodeEnd":259,"githubUrl":"https://github.com/karpathy/autoresearch/blob/228791fb499afffb54b46200aca536f79142f117/prepare.py#L223-L259","documentation":"This ValueError is raised by Tokenizer.encode (prepare.py:241) when the `text` argument is neither a str nor a list. The wrapper only supports two input shapes: a single string (encoded via encode_ordinary) or a batch, i.e. a list of strings (encoded via encode_ordinary_batch). Any other type — None, int, tuple, numpy array, torch.Tensor, dict, or a nested list — falls through the isinstance chain and hits the explicit raise, mirroring tiktoken's own input-type validation.","triggerScenarios":"Calling tokenizer.encode(None), tokenizer.encode(42), tokenizer.encode(('a','b')), or tokenizer.encode(np.array(['a'])) — anything that is not exactly `str` or exactly `list`. Also calling encode on the result of an upstream step that returned None or a tensor instead of text (e.g. a DataLoader yielding token ids instead of raw strings), or passing a generator/iterator instead of a materialized list.","commonSituations":"Feeding an already-tokenized dataset (ids) back into encode(); a data pipeline returning None on empty/missing records; numpy arrays or pandas Series values sneaking in from a parquet/DataFrame loader instead of plain Python str; passing a tuple from an unpacking; generators from lazy maps not being converted with list().","solutions":["Check what you are actually passing: log or breakpoint on `type(text)` right before the encode call and fix the producer so it emits str or list[str].","If the value is a numpy array, pandas Series, or tuple of strings, convert it explicitly: tokenizer.encode(list(texts)) or tokenizer.encode(str(texts[i])).","If the value is None, guard empty/missing records upstream (skip or substitute an empty string '') before calling encode.","If the value is a generator/iterator, materialize it first: tokenizer.encode(list(gen)).","If you meant to encode already-tokenized ids, do not call encode — decode them first or bypass the tokenizer."],"exampleFix":"// before\nids = tokenizer.encode(doc)  # doc is a numpy array / tuple / None -> ValueError\n\n// after\nif isinstance(doc, np.ndarray):\n    doc = doc.item() if doc.ndim == 0 else [str(x) for x in doc]\nelif isinstance(doc, tuple):\n    doc = list(doc)\nelif not isinstance(doc, (str, list)):\n    raise TypeError(f\"expected str or list[str], got {type(doc)!r}\")\nids = tokenizer.encode(doc)","handlingStrategy":"type-guard","validationCode":"def _is_encodable(text) -> bool:\n    if isinstance(text, str):\n        return True\n    return isinstance(text, list) and all(isinstance(t, str) for t in text)\n\nif not _is_encodable(batch):\n    raise TypeError(f\"encode expects str or list[str], got {type(batch)!r}\")\nids = tokenizer.encode(batch)","typeGuard":"from typing import Union, List\n\ndef is_encodable_input(text: object) -> bool:\n    \"\"\"Narrow to the types Tokenizer.encode accepts.\"\"\"\n    if isinstance(text, str):\n        return True\n    return isinstance(text, list) and all(isinstance(t, str) for t in text)\n\ndef encode_safe(tok, text: Union[str, List[str], None]):\n    if text is None:\n        return []\n    if not is_encodable_input(text):\n        raise TypeError(f\"expected str or list[str], got {type(text)!r}\")\n    return tok.encode(text)","tryCatchPattern":"try:\n    ids = tokenizer.encode(text)\nexcept ValueError as e:\n    if \"Invalid input type\" in str(e):\n        raise TypeError(f\"bad input to tokenizer: {type(text)!r} value={text!r:.100}\") from e\n    raise","preventionTips":["Type-annotate the data pipeline end to end so mypy flags non-str inputs before runtime.","Assert the loader contract at the boundary: assert isinstance(doc, (str, list)) after reading parquet/DataFrame records.","Convert numpy arrays and pandas values with .tolist() / str() at the source, not at the tokenizer.","Keep raw-text and token-id stages in separate, clearly named variables to avoid feeding ids back into encode."],"tags":["python","tokenizer","type-validation","valueerror","data-pipeline"],"backgroundTag":null,"analyzedSha":"228791fb499afffb54b46200aca536f79142f117","analyzedAt":"2026-08-14T19:44:21.911Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}