headroomlabs-ai/headroom · error · ValueError

Failed to load LongBench task '{task}': {e}

Error message

Failed to load LongBench task '{task}': {e}

What it means

Error "Failed to load LongBench task '{task}': {e}" thrown in headroomlabs-ai/headroom.

Source

Thrown at headroom/evals/datasets.py:406

    - narrativeqa: Story comprehension
    - gov_report: Government report summarization
    - qmsum: Meeting summarization
    - multi_news: Multi-document summarization

    Args:
        n: Number of samples to load
        task: LongBench task name

    Returns:
        EvalSuite with LongBench cases
    """
    _check_datasets_installed()
    from datasets import load_dataset

    try:
        ds = load_dataset("THUDM/LongBench", task, split="test")
    except Exception as e:
        raise ValueError(f"Failed to load LongBench task '{task}': {e}") from e

    cases: list[EvalCase] = []
    for i, item in enumerate(ds):
        if i >= n:
            break

        context = item.get("context", "")
        if not context:
            continue

        query = item.get("input", "")
        if not query:
            continue

        # Ground truth (list of answers for some tasks)
        answers = item.get("answers", [])
        ground_truth = answers[0] if answers else None

View on GitHub (pinned to 322425c43b)

Solutions

1. Check that the HuggingFace `datasets` package is installed: pip install headroom-ai[evals]
2. Verify the LongBench task name is valid and the dataset is available on the HuggingFace Hub
3. Check network connectivity and HuggingFace Hub availability; retry the download
4. If behind a proxy or offline, pre-download the dataset or set HF_DATASETS_CACHE / HF_HUB_OFFLINE appropriately

When it happens

Trigger: Raised when loading a LongBench eval task via `datasets.load_dataset` fails, typically due to a missing `datasets` dependency, an unknown task name, network failure, or Hub unavailability.

Common situations: See trigger scenarios.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/4fc9cb3356daad2f. Report an issue: GitHub.