headroomlabs-ai/headroom · error · ValueError

duplicate scenario name in suite: {built.name}

Error message

duplicate scenario name in suite: {built.name}

What it means

Raised by HeadroomSuite.add() when a HarnessScenario being added has the same name as one already registered in the suite. Scenario names are used as unique keys (e.g. for agent-evals manifest filenames like '{name}.agent-evals.json'), so duplicates are rejected to keep outputs addressable.

Source

Thrown at headroom/testing/harness.py:941

    Build = build
    Suite = suite


class HeadroomSuite:
    """Declarative matrix of built scenarios for bench and local no-key runs."""

    def __init__(self, *, name: str = "headroom-suite") -> None:
        self.name = name
        self._scenarios: list[HarnessScenario] = []

    @property
    def scenarios(self) -> tuple[HarnessScenario, ...]:
        return tuple(self._scenarios)

    def add(self, scenario: HarnessScenario | Headroom) -> HeadroomSuite:
        built = scenario.Build() if isinstance(scenario, Headroom) else scenario
        if any(existing.name == built.name for existing in self._scenarios):
            raise ValueError(f"duplicate scenario name in suite: {built.name}")
        self._scenarios.append(built)
        return self

    Add = add

    def extend(self, scenarios: Sequence[HarnessScenario | Headroom]) -> HeadroomSuite:
        for scenario in scenarios:
            self.add(scenario)
        return self

    Extend = extend

    def orchestrate(
        self,
        tasks: Sequence[ScenarioTask],
        *,
        guarantees: Sequence[Guarantee] = DEFAULT_GUARANTEES,
    ) -> ScenarioRunReport:

View on GitHub (pinned to 322425c43b)

Solutions

1. Give each scenario a distinct name before adding: `headroom.configure_scenario(name='anthropic-cached')` or whatever naming API the builder exposes.
2. Check existing names first: `if s.name in {e.name for e in suite.scenarios}: rename or skip`.
3. Dedupe the input list (e.g. by model or config key) before suite.extend().

Example fix

# before
suite.add(h1).add(h2)  # both default-named

# after
h1.configure_scenario(name="anthropic-200k")
h2.configure_scenario(name="anthropic-1m")
suite.add(h1).add(h2)

existing = {s.name for s in suite.scenarios}
if built.name in existing:
    built = built with unique name or raise/skip

try:
    suite.add(scenario)
except ValueError as e:
    if "duplicate scenario name" in str(e):
        scenario.name = f"{scenario.name}-{len(suite.scenarios)}"
        suite.add(scenario)
    else:
        raise

When it happens

Trigger: Calling suite.add(headroom) twice on builders whose .Build() produce the same default scenario name; calling suite.extend([...]) where two Headroom instances were named identically (e.g. both left as the default 'name'); re-adding a scenario after a retry loop.

Common situations: Building benchmark suites in a loop (names derived from a model list with duplicate entries); copy-pasting scenario builder blocks and forgetting to change .name; suites assembled from config files where two entries share a name key.

Related errors

• mode must be 'token' or 'cache'
• {name} is read-only; mutate its fields instead
• unknown Headroom harness config field: {name}
• unknown ProxyConfig field: {key}
• HeadroomSuite requires at least one scenario

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/81b25fcf77d92b46. Report an issue: GitHub.