headroomlabs-ai/headroom · error · ValueError

Unknown dataset: {dataset_name}

Error message

Unknown dataset: {dataset_name}

What it means

Error "Unknown dataset: {dataset_name}" thrown in headroomlabs-ai/headroom.

Source

Thrown at headroom/evals/html_oss_benchmarks.py:294

    try:
        from datasets import load_dataset
    except ImportError:
        raise ImportError("HuggingFace datasets required") from None

    if extractor is None:
        from headroom.transforms.html_extractor import HTMLExtractor

        extractor = HTMLExtractor()

    # Load QA dataset
    logger.info(f"Loading {dataset_name} dataset...")

    if dataset_name == "squad":
        dataset = load_dataset("rajpurkar/squad_v2", split="validation")
    elif dataset_name == "hotpotqa":
        dataset = load_dataset("hotpotqa/hotpot_qa", "fullwiki", split="validation")
    else:
        raise ValueError(f"Unknown dataset: {dataset_name}")

    # Select subset
    samples = dataset.select(range(min(max_questions, len(dataset))))

    logger.info(f"Evaluating {len(samples)} questions...")

    f1_original = []
    f1_extracted = []
    em_original = []
    em_extracted = []
    question_results = []

    for i, sample in enumerate(samples):
        # Get question and context
        question = sample["question"]

        if dataset_name == "squad":
            context = sample["context"]

View on GitHub (pinned to 322425c43b)

Solutions

1. Check the dataset name '{dataset_name}' for typos
2. List the supported benchmark dataset names and pick one of them
3. Upgrade headroom-ai if the dataset was added in a newer release

When it happens

Trigger: Raised when the HTML OSS benchmark runner is given a dataset name it does not recognize.

Common situations: See trigger scenarios.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/0255f7b008507c8b. Report an issue: GitHub.