karpathy/nanochat · error · ValueError

Unsupported task evaluation type: {task_object.eval_type}

Error message

Unsupported task evaluation type: {task_object.eval_type}

What it means

scripts/chat_eval.py dispatches evaluation by task_object.eval_type: 'generative' tasks (HumanEval, GSM8K) sample completions via the Engine; 'categorical' tasks (MMLU, ARC) read argmax over answer-letter logits. Any task class whose eval_type property returns something else raises ValueError. All bundled tasks in tasks/ (mmlu, arc, gsm8k, humaneval) return one of the two valid values.

Source

Thrown at scripts/chat_eval.py:174

def run_chat_eval(task_name, model, tokenizer, engine,
                   batch_size=1, num_samples=1, max_new_tokens=512, temperature=0.0, top_k=50,
                   max_problems=None):
    # Create the evaluation object
    task_module = {
        'HumanEval': HumanEval,
        'MMLU': partial(MMLU, subset="all", split="test"),
        'ARC-Easy': partial(ARC, subset="ARC-Easy", split="test"),
        'ARC-Challenge': partial(ARC, subset="ARC-Challenge", split="test"),
        'GSM8K': partial(GSM8K, subset="main", split="test"),
    }[task_name]
    task_object = task_module()
    # Run the evaluation
    if task_object.eval_type == 'generative':
        acc = run_generative_eval(task_object, tokenizer, model, engine, num_samples, max_new_tokens, temperature, top_k, max_problems=max_problems)
    elif task_object.eval_type == 'categorical':
        acc = run_categorical_eval(task_object, tokenizer, model, batch_size, max_problems=max_problems)
    else:
        raise ValueError(f"Unsupported task evaluation type: {task_object.eval_type}")
    return acc

# -----------------------------------------------------------------------------
if __name__ == "__main__":

    # Parse command-line arguments
    parser = argparse.ArgumentParser()
    parser.add_argument('-i', '--source', type=str, required=True, help="Source of the model: sft|rl")
    parser.add_argument('-a', '--task-name', type=str, default=None, help="Task name. Default = all tasks. Use | to split multiple tasks.")
    parser.add_argument('-t', '--temperature', type=float, default=0.0)
    parser.add_argument('-m', '--max-new-tokens', type=int, default=512)
    parser.add_argument('-n', '--num-samples', type=int, default=1)
    parser.add_argument('-k', '--top-k', type=int, default=50)
    parser.add_argument('-b', '--batch-size', type=int, default=8, help='Batch size for categorical evaluation')
    parser.add_argument('-g', '--model-tag', type=str, default=None, help='Model tag to load')
    parser.add_argument('-s', '--step', type=int, default=None, help='Step to load')
    parser.add_argument('-x', '--max-problems', type=int, default=None, help='Max problems to evaluate')
    parser.add_argument('--device-type', type=str, default='', choices=['cuda', 'cpu', 'mps'], help='Device type for evaluation: cuda|cpu|mps. empty => autodetect')

View on GitHub (pinned to 92d63d4e8b)

Solutions

1. Make the custom task's eval_type property return 'generative' or 'categorical' exactly.
2. Implement the matching protocol: generative tasks need __getitem__ returning a conversation and evaluate(conversation, completion); categorical tasks need conversations with 'letters' and an evaluate(conversation, letter).
3. For genuinely new eval modes, add a new elif branch with a runner function in chat_eval.py.

Example fix

# before
class MyTask:
    @property
    def eval_type(self):
        return "perplexity"

# after
class MyTask:
    @property
    def eval_type(self):
        return "categorical"

VALID_EVAL_TYPES = {'generative', 'categorical'}
task_object = task_module()
assert task_object.eval_type in VALID_EVAL_TYPES, f"task {task_name} has unsupported eval_type {task_object.eval_type!r}"

def is_supported_eval_type(eval_type) -> bool:
    return eval_type in {'generative', 'categorical'}

When it happens

Trigger: Registering a custom task class in the task_module dict whose eval_type returns an unsupported string (e.g. 'perplexity', 'loss', 'multiple_choice'), or misspelling 'generative'/'categorical' in a copied task file.

Common situations: Adding a new benchmark to tasks/ and chat_eval's registry without implementing one of the two runner protocols; renaming eval_type values across a fork; a task class missing the eval_type property entirely is a different (AttributeError) failure.

Related errors

• Unsupported task type: {task_type}
• Unknown dataset tag: {dataset_tag}
• Invalid input type: {type(text)}
• Unknown part type: {part['type']}
• Unknown content type: {type(content)}

AI-assisted analysis of karpathy/nanochat@92d63d4e8b (2026-08-15). Data as JSON: /api/errors/e95715969be51a12. Report an issue: GitHub.