headroomlabs-ai/headroom · error · ImportError

HuggingFace datasets required for this loader. Install…

Error message

HuggingFace datasets required for this loader. Install with: pip install headroom-ai[evals]

What it means

Raised by _check_datasets_installed() in headroom.evals.datasets: the HuggingFace `datasets` library is an optional dependency (shipped under the 'evals' extra), and every loader in this module (e.g. load_hotpotqa) calls the guard before importing it. The ImportError is chained from the original failure and names the exact extra to install: headroom-ai[evals].

Solutions

Install the extra in the same interpreter: `pip install 'headroom-ai[evals]'`
Or install the dependency directly: `pip install datasets`
Verify with `python -c "import datasets"` in the environment that runs the evals; fix venv mismatch if needed

Example fix

# before
from headroom.evals.datasets import load_hotpotqa
cases = load_hotpotqa(n=100)  # ImportError: HuggingFace datasets required...

# after
# pip install 'headroom-ai[evals]'
from headroom.evals.datasets import load_hotpotqa
cases = load_hotpotqa(n=100)

Defensive patterns

Strategy: validation

Validate before calling

try:
    import datasets  # noqa: F401
except ImportError:
    raise SystemExit("Eval datasets need the extra: pip install 'headroom-ai[evals]'")

Prevention

Install with the [evals] extra on machines that run dataset loaders
Add `python -c 'import datasets'` to CI for eval pipelines
Keep eval dependencies in the same interpreter as headroom to avoid venv mismatches

When it happens

Trigger: Calling any dataset loader (load_hotpotqa and siblings) with a plain `pip install headroom-ai` installation that omitted the [evals] extra, so `import datasets` fails. The guard fires before any network or parsing work.

Common situations: Minimal installs in Docker/CI for the proxy-only use case, then later running evals; stale environments from before the extra existed (datasets was perhaps a hard dep); interpreter mismatch where datasets lives in another venv.

Related errors

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/259f56a9c4c601d7. Report an issue: GitHub.

Appendix: source

Thrown at headroom/evals/datasets.py:42

Custom:
- Tool output samples: Built-in realistic tool outputs
"""

from __future__ import annotations

import json
from pathlib import Path
from typing import Any

from headroom.evals.core import EvalCase, EvalSuite


def _check_datasets_installed() -> None:
    """Check if HuggingFace datasets is installed."""
    try:
        import datasets  # noqa: F401
    except ImportError as e:
        raise ImportError(
            "HuggingFace datasets required for this loader. "
            "Install with: pip install headroom-ai[evals]"
        ) from e


# =============================================================================
# RAG / RETRIEVAL DATASETS
# =============================================================================


def load_hotpotqa(
    n: int = 100,
    split: str = "validation",
) -> EvalSuite:
    """Load HotpotQA dataset for multi-hop QA evaluation.

    HotpotQA contains questions requiring reasoning over multiple
    Wikipedia passages, with verified ground truth answers.

View on GitHub (pinned to 322425c43b)