headroomlabs-ai/headroom · error · ValueError

Failed to download BFCL dataset '{data_file}': {e}

Error message

Failed to download BFCL dataset '{data_file}': {e}

What it means

Error "Failed to download BFCL dataset '{data_file}': {e}" thrown in headroomlabs-ai/headroom.

Source

Thrown at headroom/evals/datasets.py:601

    Args:
        n: Number of samples to load
        category: BFCL category

    Returns:
        EvalSuite with BFCL cases
    """
    import urllib.request

    base_url = "https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard/resolve/main"
    data_file = f"BFCL_v3_{category}.json"
    gt_file = f"possible_answer/BFCL_v3_{category}.json"

    # Download questions + function schemas (JSONL)
    try:
        raw = urllib.request.urlopen(f"{base_url}/{data_file}").read().decode("utf-8")  # nosec B310
        items = [json.loads(line) for line in raw.strip().split("\n") if line.strip()]
    except Exception as e:
        raise ValueError(f"Failed to download BFCL dataset '{data_file}': {e}") from e

    # Download ground truth (JSONL, keyed by id)
    gt_by_id: dict[str, str] = {}
    try:
        gt_raw = urllib.request.urlopen(f"{base_url}/{gt_file}").read().decode("utf-8")  # nosec B310
        for line in gt_raw.strip().split("\n"):
            if line.strip():
                obj = json.loads(line)
                gt_by_id[obj["id"]] = json.dumps(obj.get("ground_truth", []))
    except Exception:
        pass  # Ground truth is optional

    cases: list[EvalCase] = []
    for i, item in enumerate(items):
        if i >= n:
            break

        item_id = item.get("id", f"bfcl_{category}_{i}")

View on GitHub (pinned to 322425c43b)

Solutions

1. Check network connectivity and retry the BFCL dataset download
2. Verify the BFCL data file URL/path '{data_file}' is correct and still published
3. If the download is blocked, download the file manually and place it in the expected cache directory
4. Inspect the underlying exception {e} for HTTP status details (403/404 indicate a moved or gated file)

When it happens

Trigger: Raised when downloading a BFCL (Berkeley Function Calling Leaderboard) dataset file fails, typically a network error, moved URL, or blocked request.

Common situations: See trigger scenarios.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/cac3b94c2999302f. Report an issue: GitHub.