headroomlabs-ai/headroom · error · ValueError

Failed to load multi-wiki-qa '{lang}': {e}

Error message

Failed to load multi-wiki-qa '{lang}': {e}

What it means

Error "Failed to load multi-wiki-qa '{lang}': {e}" thrown in headroomlabs-ai/headroom.

Source

Thrown at headroom/evals/datasets.py:468

    zh/ja/ko extractive QA, so it yields comparable cross-language compression
    answer-retention numbers. License: CC-BY-NC-SA-4.0 (non-commercial).

    Common lang configs: "zh-cn", "ja", "ko".

    Args:
        n: Number of samples to load
        lang: multi-wiki-qa language config (e.g. "ja", "ko", "zh-cn")

    Returns:
        EvalSuite with multi-wiki-qa cases
    """
    _check_datasets_installed()
    from datasets import load_dataset

    try:
        ds = load_dataset("alexandrainst/multi-wiki-qa", lang, split=f"train[:{n}]")
    except Exception as e:
        raise ValueError(f"Failed to load multi-wiki-qa '{lang}': {e}") from e

    cases: list[EvalCase] = []
    for i, item in enumerate(ds):
        context = item.get("context", "")
        question = item.get("question", "")
        answers = item.get("answers") or {}
        texts = answers.get("text") if isinstance(answers, dict) else None
        ground_truth = texts[0] if texts else None
        if not (context and question and ground_truth):
            continue

        cases.append(
            EvalCase(
                id=f"multi_wiki_qa_{lang}_{i}",
                context=context,
                query=question,
                ground_truth=ground_truth,
                metadata={

View on GitHub (pinned to 322425c43b)

Solutions

1. Install the HuggingFace datasets package: pip install headroom-ai[evals]
2. Verify the requested language code '{lang}' is supported by the multi-wiki-qa dataset
3. Check network connectivity to the HuggingFace Hub and retry
4. Pre-download the dataset into HF_DATASETS_CACHE if running offline

When it happens

Trigger: Raised when loading the multi-wiki-qa eval dataset for a given language fails, due to a missing `datasets` dependency, unsupported language, or network/Hub errors.

Common situations: See trigger scenarios.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/1c6b8b78947419e3. Report an issue: GitHub.