{"record":{"id":"4fc9cb3356daad2f","repo":"headroomlabs-ai/headroom","slug":"failed-to-load-longbench-task-task-e","errorCode":null,"errorMessage":"Failed to load LongBench task '{task}': {e}","messagePattern":"Failed to load LongBench task '(.+?)': (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"headroom/evals/datasets.py","lineNumber":406,"sourceCode":"    - narrativeqa: Story comprehension\n    - gov_report: Government report summarization\n    - qmsum: Meeting summarization\n    - multi_news: Multi-document summarization\n\n    Args:\n        n: Number of samples to load\n        task: LongBench task name\n\n    Returns:\n        EvalSuite with LongBench cases\n    \"\"\"\n    _check_datasets_installed()\n    from datasets import load_dataset\n\n    try:\n        ds = load_dataset(\"THUDM/LongBench\", task, split=\"test\")\n    except Exception as e:\n        raise ValueError(f\"Failed to load LongBench task '{task}': {e}\") from e\n\n    cases: list[EvalCase] = []\n    for i, item in enumerate(ds):\n        if i >= n:\n            break\n\n        context = item.get(\"context\", \"\")\n        if not context:\n            continue\n\n        query = item.get(\"input\", \"\")\n        if not query:\n            continue\n\n        # Ground truth (list of answers for some tasks)\n        answers = item.get(\"answers\", [])\n        ground_truth = answers[0] if answers else None\n","sourceCodeStart":388,"sourceCodeEnd":424,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/evals/datasets.py#L388-L424","documentation":"Error \"Failed to load LongBench task '{task}': {e}\" thrown in headroomlabs-ai/headroom.","triggerScenarios":"Raised when loading a LongBench eval task via `datasets.load_dataset` fails, typically due to a missing `datasets` dependency, an unknown task name, network failure, or Hub unavailability.","commonSituations":"See trigger scenarios.","solutions":["Check that the HuggingFace `datasets` package is installed: pip install headroom-ai[evals]","Verify the LongBench task name is valid and the dataset is available on the HuggingFace Hub","Check network connectivity and HuggingFace Hub availability; retry the download","If behind a proxy or offline, pre-download the dataset or set HF_DATASETS_CACHE / HF_HUB_OFFLINE appropriately"],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}