headroomlabs-ai/headroom · error · ImportError

HuggingFace datasets required

Error message

HuggingFace datasets required

What it means

Error "HuggingFace datasets required" thrown in headroomlabs-ai/headroom.

Source

Thrown at headroom/evals/html_oss_benchmarks.py:279

) -> QAAccuracyResult:
    """Evaluate whether HTML extraction preserves QA accuracy.

    This test verifies that LLMs can answer questions equally well
    (or better) from extracted content vs original HTML.

    Args:
        answer_fn: Function(context, question) -> answer string
        extractor: HTMLExtractor instance
        max_questions: Number of questions to evaluate
        dataset_name: Which dataset to use ("squad" or "hotpotqa")

    Returns:
        QAAccuracyResult showing whether accuracy is preserved
    """
    try:
        from datasets import load_dataset
    except ImportError:
        raise ImportError("HuggingFace datasets required") from None

    if extractor is None:
        from headroom.transforms.html_extractor import HTMLExtractor

        extractor = HTMLExtractor()

    # Load QA dataset
    logger.info(f"Loading {dataset_name} dataset...")

    if dataset_name == "squad":
        dataset = load_dataset("rajpurkar/squad_v2", split="validation")
    elif dataset_name == "hotpotqa":
        dataset = load_dataset("hotpotqa/hotpot_qa", "fullwiki", split="validation")
    else:
        raise ValueError(f"Unknown dataset: {dataset_name}")

    # Select subset
    samples = dataset.select(range(min(max_questions, len(dataset))))

View on GitHub (pinned to 322425c43b)

Solutions

1. Install the datasets package: pip install datasets
2. Or install the evals extra: pip install headroom-ai[evals]

When it happens

Trigger: Raised when an HTML OSS benchmark requires the HuggingFace `datasets` library and it is missing from the environment.

Common situations: See trigger scenarios.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/cb7e4cf2c1050a9a. Report an issue: GitHub.