headroomlabs-ai/headroom · error · ValueError

ScenarioOrchestrator requires at least one scenario

Error message

ScenarioOrchestrator requires at least one scenario

What it means

Raised by ScenarioOrchestrator.__init__ when constructed with an empty scenarios sequence. The orchestrator's job is to cross scenarios with tasks and evaluate guarantees, so an empty scenario list is treated as a caller bug and rejected immediately rather than producing an empty report.

Source

Thrown at headroom/testing/harness.py:1524

            time.sleep(0.25)
        raise TimeoutError(f"headroom proxy was not ready at {url}: {last_error}")


class _NoopClient:
    """Minimal original-client object for SDK simulations without upstream I/O."""


class ScenarioOrchestrator:
    """Runs built scenarios against local no-key tasks and evaluates guarantees."""

    def __init__(
        self,
        scenarios: Sequence[HarnessScenario],
        *,
        guarantees: Sequence[Guarantee] = DEFAULT_GUARANTEES,
    ) -> None:
        if not scenarios:
            raise ValueError("ScenarioOrchestrator requires at least one scenario")
        self._scenarios = tuple(scenarios)
        self._guarantees = tuple(guarantees)

    def run(self, tasks: Sequence[ScenarioTask]) -> ScenarioRunReport:
        if not tasks:
            raise ValueError("ScenarioOrchestrator.run requires at least one task")

        cases: list[ScenarioCaseResult] = []
        for scenario in self._scenarios:
            for task in tasks:
                result = scenario.simulate(
                    task.messages,
                    model=task.model,
                    provider=task.provider,
                    output_buffer_tokens=task.output_buffer_tokens,
                )
                guarantees = tuple(
                    guarantee(scenario, task, result) for guarantee in self._guarantees

View on GitHub (pinned to 322425c43b)

Solutions

1. Pass at least one scenario: orchestrator = ScenarioOrchestrator([scenario]) or ScenarioOrchestrator(suite.scenarios) after suite.add(...).
2. If scenarios are dynamic, assert non-empty before constructing: `assert scenarios, 'no scenarios selected'`.
3. Check the filter/env var that produced the empty sequence.

Example fix

# before
orch = ScenarioOrchestrator([])  # ValueError

# after
orch = ScenarioOrchestrator([scenario_a, scenario_b])

assert scenarios, "ScenarioOrchestrator needs >=1 scenario"
orch = ScenarioOrchestrator(scenarios)

When it happens

Trigger: ScenarioOrchestrator([]) or ScenarioOrchestrator(suite.scenarios) where the suite had no scenarios; passing a filtered/generator-derived sequence that yielded nothing; constructing the orchestrator before adding scenarios to the suite and never re-creating it.

Common situations: Parameterized test configs selecting zero scenarios via env filter; refactors converting a list comprehension to a generator consumed elsewhere; suite built but scenarios added after orchestrator creation.

Related errors

• HeadroomSuite requires at least one scenario
• ScenarioOrchestrator.run requires at least one task
• mode must be 'token' or 'cache'
• invalid pipeline config TOML: {0}
• recommendations file not found: {0}

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/f5f5fb356021e4f2. Report an issue: GitHub.