{"record":{"id":"e95715969be51a12","repo":"karpathy/nanochat","slug":"unsupported-task-evaluation-type-task-object-eva","errorCode":null,"errorMessage":"Unsupported task evaluation type: {task_object.eval_type}","messagePattern":"Unsupported task evaluation type: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scripts/chat_eval.py","lineNumber":174,"sourceCode":"def run_chat_eval(task_name, model, tokenizer, engine,\n                   batch_size=1, num_samples=1, max_new_tokens=512, temperature=0.0, top_k=50,\n                   max_problems=None):\n    # Create the evaluation object\n    task_module = {\n        'HumanEval': HumanEval,\n        'MMLU': partial(MMLU, subset=\"all\", split=\"test\"),\n        'ARC-Easy': partial(ARC, subset=\"ARC-Easy\", split=\"test\"),\n        'ARC-Challenge': partial(ARC, subset=\"ARC-Challenge\", split=\"test\"),\n        'GSM8K': partial(GSM8K, subset=\"main\", split=\"test\"),\n    }[task_name]\n    task_object = task_module()\n    # Run the evaluation\n    if task_object.eval_type == 'generative':\n        acc = run_generative_eval(task_object, tokenizer, model, engine, num_samples, max_new_tokens, temperature, top_k, max_problems=max_problems)\n    elif task_object.eval_type == 'categorical':\n        acc = run_categorical_eval(task_object, tokenizer, model, batch_size, max_problems=max_problems)\n    else:\n        raise ValueError(f\"Unsupported task evaluation type: {task_object.eval_type}\")\n    return acc\n\n# -----------------------------------------------------------------------------\nif __name__ == \"__main__\":\n\n    # Parse command-line arguments\n    parser = argparse.ArgumentParser()\n    parser.add_argument('-i', '--source', type=str, required=True, help=\"Source of the model: sft|rl\")\n    parser.add_argument('-a', '--task-name', type=str, default=None, help=\"Task name. Default = all tasks. Use | to split multiple tasks.\")\n    parser.add_argument('-t', '--temperature', type=float, default=0.0)\n    parser.add_argument('-m', '--max-new-tokens', type=int, default=512)\n    parser.add_argument('-n', '--num-samples', type=int, default=1)\n    parser.add_argument('-k', '--top-k', type=int, default=50)\n    parser.add_argument('-b', '--batch-size', type=int, default=8, help='Batch size for categorical evaluation')\n    parser.add_argument('-g', '--model-tag', type=str, default=None, help='Model tag to load')\n    parser.add_argument('-s', '--step', type=int, default=None, help='Step to load')\n    parser.add_argument('-x', '--max-problems', type=int, default=None, help='Max problems to evaluate')\n    parser.add_argument('--device-type', type=str, default='', choices=['cuda', 'cpu', 'mps'], help='Device type for evaluation: cuda|cpu|mps. empty => autodetect')","sourceCodeStart":156,"sourceCodeEnd":192,"githubUrl":"https://github.com/karpathy/nanochat/blob/92d63d4e8bb4df75c3b71618f31ddde2378b2bcd/scripts/chat_eval.py#L156-L192","documentation":"scripts/chat_eval.py dispatches evaluation by task_object.eval_type: 'generative' tasks (HumanEval, GSM8K) sample completions via the Engine; 'categorical' tasks (MMLU, ARC) read argmax over answer-letter logits. Any task class whose eval_type property returns something else raises ValueError. All bundled tasks in tasks/ (mmlu, arc, gsm8k, humaneval) return one of the two valid values.","triggerScenarios":"Registering a custom task class in the task_module dict whose eval_type returns an unsupported string (e.g. 'perplexity', 'loss', 'multiple_choice'), or misspelling 'generative'/'categorical' in a copied task file.","commonSituations":"Adding a new benchmark to tasks/ and chat_eval's registry without implementing one of the two runner protocols; renaming eval_type values across a fork; a task class missing the eval_type property entirely is a different (AttributeError) failure.","solutions":["Make the custom task's eval_type property return 'generative' or 'categorical' exactly.","Implement the matching protocol: generative tasks need __getitem__ returning a conversation and evaluate(conversation, completion); categorical tasks need conversations with 'letters' and an evaluate(conversation, letter).","For genuinely new eval modes, add a new elif branch with a runner function in chat_eval.py."],"exampleFix":"# before\nclass MyTask:\n    @property\n    def eval_type(self):\n        return \"perplexity\"\n\n# after\nclass MyTask:\n    @property\n    def eval_type(self):\n        return \"categorical\"","handlingStrategy":"validation","validationCode":"VALID_EVAL_TYPES = {'generative', 'categorical'}\ntask_object = task_module()\nassert task_object.eval_type in VALID_EVAL_TYPES, f\"task {task_name} has unsupported eval_type {task_object.eval_type!r}\"","typeGuard":"def is_supported_eval_type(eval_type) -> bool:\n    return eval_type in {'generative', 'categorical'}","tryCatchPattern":null,"preventionTips":["When adding a task class, copy an existing task (arc.py or gsm8k.py) so eval_type matches a supported value.","Run chat_eval on a single task with --max-problems 2 first to smoke-test custom tasks.","Implement the full protocol (conversation shape, evaluate()) matching the chosen eval_type."],"tags":["nanochat","evaluation","chat-eval","validation"],"backgroundTag":null,"analyzedSha":"92d63d4e8bb4df75c3b71618f31ddde2378b2bcd","analyzedAt":"2026-08-15T03:11:54.371Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}