{"record":{"id":"2dae8f3e17fb38f8","repo":"karpathy/nanochat","slug":"unsupported-task-type-task-type","errorCode":null,"errorMessage":"Unsupported task type: {task_type}","messagePattern":"Unsupported task type: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"nanochat/core_eval.py","lineNumber":194,"sourceCode":"    fewshot_examples = []\n    if num_fewshot > 0:\n        rng = random.Random(1234 + idx)\n        available_indices = [i for i in range(len(data)) if i != idx]\n        fewshot_indices = rng.sample(available_indices, num_fewshot)\n        fewshot_examples = [data[i] for i in fewshot_indices]\n\n    # Render prompts and batch sequences based on task type\n    if task_type == 'multiple_choice':\n        prompts = render_prompts_mc(item, continuation_delimiter, fewshot_examples)\n        tokens, start_idxs, end_idxs = batch_sequences_mc(tokenizer, prompts)\n    elif task_type == 'schema':\n        prompts = render_prompts_schema(item, continuation_delimiter, fewshot_examples)\n        tokens, start_idxs, end_idxs = batch_sequences_schema(tokenizer, prompts)\n    elif task_type == 'language_modeling':\n        prompts = render_prompts_lm(item, continuation_delimiter, fewshot_examples)\n        tokens, start_idxs, end_idxs = batch_sequences_lm(tokenizer, prompts)\n    else:\n        raise ValueError(f\"Unsupported task type: {task_type}\")\n\n    # Some models can't forward sequences beyond a certain length (e.g. GPT-2)\n    # In these cases, we have to truncate sequences to max length and adjust the indices\n    if hasattr(model, 'max_seq_len') and model.max_seq_len is not None:\n        max_tokens = model.max_seq_len\n        new_tokens, new_start_idxs, new_end_idxs = [], [], []\n        for t, s, e in zip(tokens, start_idxs, end_idxs):\n            if len(t) > max_tokens:\n                num_to_crop = len(t) - max_tokens\n                new_tokens.append(t[-max_tokens:]) # take the last max_tokens tokens\n                new_start_idxs.append(s - num_to_crop) # shift the indices down\n                new_end_idxs.append(e - num_to_crop)\n                assert s - num_to_crop >= 0, \"this should never happen right?\"\n                assert e - num_to_crop >= 0, \"this should never happen right?\"\n            else:\n                new_tokens.append(t) # keep unchanged\n                new_start_idxs.append(s)\n                new_end_idxs.append(e)","sourceCodeStart":176,"sourceCodeEnd":212,"githubUrl":"https://github.com/karpathy/nanochat/blob/92d63d4e8bb4df75c3b71618f31ddde2378b2bcd/nanochat/core_eval.py#L176-L212","documentation":"In `evaluate_example` (core_eval.py), the `task_type` field of the task metadata dict must be one of 'multiple_choice', 'schema', or 'language_modeling'; it selects the prompt renderer and sequence batching strategy. Any other value raises ValueError before the model is ever called.","triggerScenarios":"Running CORE evaluation with a task whose metadata declares a task_type outside the supported set — e.g. a custom/added task with task_type 'generative', 'cloze', or a typo like 'multiplechoice'.","commonSituations":"Adding a new benchmark to the core eval task registry without implementing a renderer; typo in the task_meta dict; copying a task definition from another eval framework with different type names.","solutions":["Set task_meta['task_type'] to exactly 'multiple_choice', 'schema', or 'language_modeling'.","For a genuinely new task type, add a matching elif branch (render + batch functions) in core_eval.py before the else.","Print/inspect the task metadata to catch typos before dispatch."],"exampleFix":"# before\ntask_meta = {'task_type': 'multiplechoice', ...}\n\n# after\ntask_meta = {'task_type': 'multiple_choice', ...}","handlingStrategy":"validation","validationCode":"SUPPORTED_TASK_TYPES = {'multiple_choice', 'schema', 'language_modeling'}\nassert task_meta['task_type'] in SUPPORTED_TASK_TYPES, f\"task_type must be one of {SUPPORTED_TASK_TYPES}, got {task_meta['task_type']!r}\"","typeGuard":"def is_supported_task_type(task_type) -> bool:\n    return task_type in {'multiple_choice', 'schema', 'language_modeling'}","tryCatchPattern":null,"preventionTips":["Validate task_meta['task_type'] against a constant set when building the task registry.","When adding a new task type, extend both dispatch sites and the renderer/batcher together."],"tags":["nanochat","evaluation","core-eval","validation"],"backgroundTag":null,"analyzedSha":"92d63d4e8bb4df75c3b71618f31ddde2378b2bcd","analyzedAt":"2026-08-15T03:11:54.371Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}