headroomlabs-ai/headroom · error · ValueError

Unknown dataset '{name}'. Available: {available}

Error message

Unknown dataset '{name}'. Available: {available}

What it means

Error "Unknown dataset '{name}'. Available: {available}" thrown in headroomlabs-ai/headroom.

Source

Thrown at headroom/evals/datasets.py:1322

def load_dataset_by_name(
    name: str,
    n: int | None = None,
    **kwargs: Any,
) -> EvalSuite:
    """Load a dataset by name from the registry.

    Args:
        name: Dataset name from registry
        n: Number of samples (uses default if not specified)
        **kwargs: Additional arguments for the loader

    Returns:
        EvalSuite with loaded cases
    """
    if name not in DATASET_REGISTRY:
        available = ", ".join(DATASET_REGISTRY.keys())
        raise ValueError(f"Unknown dataset '{name}'. Available: {available}")

    info = DATASET_REGISTRY[name]
    loader = info["loader"]

    # Use provided n or default
    if n is None:
        n = info.get("default_n")

    if n is not None:
        result: EvalSuite = loader(n=n, **kwargs)
    else:
        result = loader(**kwargs)
    return result

View on GitHub (pinned to 322425c43b)

Solutions

1. Pick a dataset name from the available list: {available}
2. Check for typos in the dataset name '{name}'
3. If the dataset was recently added, upgrade headroom-ai to a version that includes it

When it happens

Trigger: Raised when an eval dataset name passed by the user is not in the registry of known datasets.

Common situations: See trigger scenarios.

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/22268330c5674b4c. Report an issue: GitHub.