headroomlabs-ai/headroom · error · ImportError

HuggingFace datasets required for this loader. Install with:

Error message

HuggingFace datasets required for this loader. Install with: pip install headroom-ai[evals]

What it means

Raised by _check_datasets_installed() in headroom.evals.datasets: the HuggingFace `datasets` library is an optional dependency (shipped under the 'evals' extra), and every loader in this module (e.g. load_hotpotqa) calls the guard before importing it. The ImportError is chained from the original failure and names the exact extra to install: headroom-ai[evals].

Source

Thrown at headroom/evals/datasets.py:42

Custom:
- Tool output samples: Built-in realistic tool outputs
"""

from __future__ import annotations

import json
from pathlib import Path
from typing import Any

from headroom.evals.core import EvalCase, EvalSuite


def _check_datasets_installed() -> None:
    """Check if HuggingFace datasets is installed."""
    try:
        import datasets  # noqa: F401
    except ImportError as e:
        raise ImportError(
            "HuggingFace datasets required for this loader. "
            "Install with: pip install headroom-ai[evals]"
        ) from e


# =============================================================================
# RAG / RETRIEVAL DATASETS
# =============================================================================


def load_hotpotqa(
    n: int = 100,
    split: str = "validation",
) -> EvalSuite:
    """Load HotpotQA dataset for multi-hop QA evaluation.

    HotpotQA contains questions requiring reasoning over multiple
    Wikipedia passages, with verified ground truth answers.

View on GitHub (pinned to 322425c43b)

Solutions

1. Install the extra in the same interpreter: `pip install 'headroom-ai[evals]'`
2. Or install the dependency directly: `pip install datasets`
3. Verify with `python -c "import datasets"` in the environment that runs the evals; fix venv mismatch if needed

Example fix

# before
from headroom.evals.datasets import load_hotpotqa
cases = load_hotpotqa(n=100)  # ImportError: HuggingFace datasets required...

# after
# pip install 'headroom-ai[evals]'
from headroom.evals.datasets import load_hotpotqa
cases = load_hotpotqa(n=100)

try:
    import datasets  # noqa: F401
except ImportError:
    raise SystemExit("Eval datasets need the extra: pip install 'headroom-ai[evals]'")

When it happens

Trigger: Calling any dataset loader (load_hotpotqa and siblings) with a plain `pip install headroom-ai` installation that omitted the [evals] extra, so `import datasets` fails. The guard fires before any network or parsing work.

Common situations: Minimal installs in Docker/CI for the proxy-only use case, then later running evals; stale environments from before the extra existed (datasets was perhaps a hard dep); interpreter mismatch where datasets lives in another venv.

Related errors

• anthropic package required. Install with: pip install anthro
• openai package required. Install with: pip install openai
• Magika is required for ML-based content detection. Install w
• openai package not installed. Install with: pip install open
• qdrant-client not installed. Install with: pip install qdran

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/259f56a9c4c601d7. Report an issue: GitHub.