karpathy/autoresearch · error · ValueError

Invalid input type: {type(text)}

Error message

Invalid input type: {type(text)}

What it means

This ValueError is raised by Tokenizer.encode (prepare.py:241) when the `text` argument is neither a str nor a list. The wrapper only supports two input shapes: a single string (encoded via encode_ordinary) or a batch, i.e. a list of strings (encoded via encode_ordinary_batch). Any other type — None, int, tuple, numpy array, torch.Tensor, dict, or a nested list — falls through the isinstance chain and hits the explicit raise, mirroring tiktoken's own input-type validation.

Source

Thrown at prepare.py:241

        return self.enc.n_vocab

    def get_bos_token_id(self):
        return self.bos_token_id

    def encode(self, text, prepend=None, num_threads=8):
        if prepend is not None:
            prepend_id = prepend if isinstance(prepend, int) else self.enc.encode_single_token(prepend)
        if isinstance(text, str):
            ids = self.enc.encode_ordinary(text)
            if prepend is not None:
                ids.insert(0, prepend_id)
        elif isinstance(text, list):
            ids = self.enc.encode_ordinary_batch(text, num_threads=num_threads)
            if prepend is not None:
                for row in ids:
                    row.insert(0, prepend_id)
        else:
            raise ValueError(f"Invalid input type: {type(text)}")
        return ids

    def decode(self, ids):
        return self.enc.decode(ids)


def get_token_bytes(device="cpu"):
    path = os.path.join(TOKENIZER_DIR, "token_bytes.pt")
    with open(path, "rb") as f:
        return torch.load(f, map_location=device)


def _document_batches(split, tokenizer_batch_size=128):
    """Infinite iterator over document batches from parquet files."""
    parquet_paths = list_parquet_files()
    assert len(parquet_paths) > 0, "No parquet files found. Run prepare.py first."
    val_path = os.path.join(DATA_DIR, VAL_FILENAME)
    if split == "train":

View on GitHub (pinned to 228791fb49)

Solutions

1. Check what you are actually passing: log or breakpoint on `type(text)` right before the encode call and fix the producer so it emits str or list[str].
2. If the value is a numpy array, pandas Series, or tuple of strings, convert it explicitly: tokenizer.encode(list(texts)) or tokenizer.encode(str(texts[i])).
3. If the value is None, guard empty/missing records upstream (skip or substitute an empty string '') before calling encode.
4. If the value is a generator/iterator, materialize it first: tokenizer.encode(list(gen)).
5. If you meant to encode already-tokenized ids, do not call encode — decode them first or bypass the tokenizer.

Example fix

// before
ids = tokenizer.encode(doc)  # doc is a numpy array / tuple / None -> ValueError

// after
if isinstance(doc, np.ndarray):
    doc = doc.item() if doc.ndim == 0 else [str(x) for x in doc]
elif isinstance(doc, tuple):
    doc = list(doc)
elif not isinstance(doc, (str, list)):
    raise TypeError(f"expected str or list[str], got {type(doc)!r}")
ids = tokenizer.encode(doc)

def _is_encodable(text) -> bool:
    if isinstance(text, str):
        return True
    return isinstance(text, list) and all(isinstance(t, str) for t in text)

if not _is_encodable(batch):
    raise TypeError(f"encode expects str or list[str], got {type(batch)!r}")
ids = tokenizer.encode(batch)

from typing import Union, List

def is_encodable_input(text: object) -> bool:
    """Narrow to the types Tokenizer.encode accepts."""
    if isinstance(text, str):
        return True
    return isinstance(text, list) and all(isinstance(t, str) for t in text)

def encode_safe(tok, text: Union[str, List[str], None]):
    if text is None:
        return []
    if not is_encodable_input(text):
        raise TypeError(f"expected str or list[str], got {type(text)!r}")
    return tok.encode(text)

try:
    ids = tokenizer.encode(text)
except ValueError as e:
    if "Invalid input type" in str(e):
        raise TypeError(f"bad input to tokenizer: {type(text)!r} value={text!r:.100}") from e
    raise

When it happens

Trigger: Calling tokenizer.encode(None), tokenizer.encode(42), tokenizer.encode(('a','b')), or tokenizer.encode(np.array(['a'])) — anything that is not exactly `str` or exactly `list`. Also calling encode on the result of an upstream step that returned None or a tensor instead of text (e.g. a DataLoader yielding token ids instead of raw strings), or passing a generator/iterator instead of a materialized list.

Common situations: Feeding an already-tokenized dataset (ids) back into encode(); a data pipeline returning None on empty/missing records; numpy arrays or pandas Series values sneaking in from a parquet/DataFrame loader instead of plain Python str; passing a tuple from an unpacking; generators from lazy maps not being converted with list().

AI-assisted analysis of karpathy/autoresearch@228791fb49 (2026-08-14). Data as JSON: /api/errors/5f99ff82e6c2fd7b. Report an issue: GitHub.