karpathy/nanochat · error · ValueError

Unsupported task type: {task_type}

Error message

Unsupported task type: {task_type}

What it means

In `evaluate_example` (core_eval.py), the `task_type` field of the task metadata dict must be one of 'multiple_choice', 'schema', or 'language_modeling'; it selects the prompt renderer and sequence batching strategy. Any other value raises ValueError before the model is ever called.

Source

Thrown at nanochat/core_eval.py:194

    fewshot_examples = []
    if num_fewshot > 0:
        rng = random.Random(1234 + idx)
        available_indices = [i for i in range(len(data)) if i != idx]
        fewshot_indices = rng.sample(available_indices, num_fewshot)
        fewshot_examples = [data[i] for i in fewshot_indices]

    # Render prompts and batch sequences based on task type
    if task_type == 'multiple_choice':
        prompts = render_prompts_mc(item, continuation_delimiter, fewshot_examples)
        tokens, start_idxs, end_idxs = batch_sequences_mc(tokenizer, prompts)
    elif task_type == 'schema':
        prompts = render_prompts_schema(item, continuation_delimiter, fewshot_examples)
        tokens, start_idxs, end_idxs = batch_sequences_schema(tokenizer, prompts)
    elif task_type == 'language_modeling':
        prompts = render_prompts_lm(item, continuation_delimiter, fewshot_examples)
        tokens, start_idxs, end_idxs = batch_sequences_lm(tokenizer, prompts)
    else:
        raise ValueError(f"Unsupported task type: {task_type}")

    # Some models can't forward sequences beyond a certain length (e.g. GPT-2)
    # In these cases, we have to truncate sequences to max length and adjust the indices
    if hasattr(model, 'max_seq_len') and model.max_seq_len is not None:
        max_tokens = model.max_seq_len
        new_tokens, new_start_idxs, new_end_idxs = [], [], []
        for t, s, e in zip(tokens, start_idxs, end_idxs):
            if len(t) > max_tokens:
                num_to_crop = len(t) - max_tokens
                new_tokens.append(t[-max_tokens:]) # take the last max_tokens tokens
                new_start_idxs.append(s - num_to_crop) # shift the indices down
                new_end_idxs.append(e - num_to_crop)
                assert s - num_to_crop >= 0, "this should never happen right?"
                assert e - num_to_crop >= 0, "this should never happen right?"
            else:
                new_tokens.append(t) # keep unchanged
                new_start_idxs.append(s)
                new_end_idxs.append(e)

View on GitHub (pinned to 92d63d4e8b)

Solutions

1. Set task_meta['task_type'] to exactly 'multiple_choice', 'schema', or 'language_modeling'.
2. For a genuinely new task type, add a matching elif branch (render + batch functions) in core_eval.py before the else.
3. Print/inspect the task metadata to catch typos before dispatch.

Example fix

# before
task_meta = {'task_type': 'multiplechoice', ...}

# after
task_meta = {'task_type': 'multiple_choice', ...}

SUPPORTED_TASK_TYPES = {'multiple_choice', 'schema', 'language_modeling'}
assert task_meta['task_type'] in SUPPORTED_TASK_TYPES, f"task_type must be one of {SUPPORTED_TASK_TYPES}, got {task_meta['task_type']!r}"

def is_supported_task_type(task_type) -> bool:
    return task_type in {'multiple_choice', 'schema', 'language_modeling'}

When it happens

Trigger: Running CORE evaluation with a task whose metadata declares a task_type outside the supported set — e.g. a custom/added task with task_type 'generative', 'cloze', or a typo like 'multiplechoice'.

Common situations: Adding a new benchmark to the core eval task registry without implementing a renderer; typo in the task_meta dict; copying a task definition from another eval framework with different type names.

Related errors

• Unsupported task evaluation type: {task_object.eval_type}
• Unknown dataset tag: {dataset_tag}
• Invalid input type: {type(text)}
• Unknown part type: {part['type']}
• Unknown content type: {type(content)}

AI-assisted analysis of karpathy/nanochat@92d63d4e8b (2026-08-15). Data as JSON: /api/errors/2dae8f3e17fb38f8. Report an issue: GitHub.