headroomlabs-ai/headroom · error · ValueError

Failed to load ToolBench category '{category}': {e}

Error message

Failed to load ToolBench category '{category}': {e}

What it means

Error "Failed to load ToolBench category '{category}': {e}" thrown in headroomlabs-ai/headroom.

Source

Thrown at headroom/evals/datasets.py:684

    Categories:
    - G1: Single-tool single-step
    - G2: Single-tool multi-step
    - G3: Multi-tool single-step

    Args:
        n: Number of samples to load
        category: ToolBench category (G1, G2, G3)

    Returns:
        EvalSuite with ToolBench cases
    """
    _check_datasets_installed()
    from datasets import load_dataset

    try:
        ds = load_dataset("ToolBench/ToolBench", category, split="test")
    except Exception as e:
        raise ValueError(f"Failed to load ToolBench category '{category}': {e}") from e

    cases: list[EvalCase] = []
    for i, item in enumerate(ds):
        if len(cases) >= n:
            break

        # Get tool definitions as context
        tools = item.get("api_list", [])
        if not tools:
            continue

        # Format tools as JSON context
        tool_defs = []
        for tool in tools:
            tool_defs.append(
                {
                    "name": tool.get("api_name", ""),
                    "description": tool.get("api_description", ""),

View on GitHub (pinned to 322425c43b)

Solutions

1. Verify the ToolBench category '{category}' is spelled correctly and exists in the dataset
2. Check that the ToolBench data files are present/downloaded in the expected location
3. Check network connectivity if the loader fetches data remotely
4. Inspect the underlying error {e} for the specific failure (missing file, parse error, network)

When it happens

Trigger: Raised when loading a ToolBench category fails, due to an unknown category name, missing local data files, or a network/parse failure.

Common situations: See trigger scenarios.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/f375a692565ad972. Report an issue: GitHub.