headroomlabs-ai/headroom · error · ValueError

ScenarioOrchestrator.run requires at least one task

Error message

ScenarioOrchestrator.run requires at least one task

What it means

Raised by ScenarioOrchestrator.run() when the tasks argument is an empty sequence. run() cross-products scenarios x tasks to produce ScenarioCaseResults; with no tasks there is nothing to evaluate, so it fails fast with ValueError.

Source

Thrown at headroom/testing/harness.py:1530

class ScenarioOrchestrator:
    """Runs built scenarios against local no-key tasks and evaluates guarantees."""

    def __init__(
        self,
        scenarios: Sequence[HarnessScenario],
        *,
        guarantees: Sequence[Guarantee] = DEFAULT_GUARANTEES,
    ) -> None:
        if not scenarios:
            raise ValueError("ScenarioOrchestrator requires at least one scenario")
        self._scenarios = tuple(scenarios)
        self._guarantees = tuple(guarantees)

    def run(self, tasks: Sequence[ScenarioTask]) -> ScenarioRunReport:
        if not tasks:
            raise ValueError("ScenarioOrchestrator.run requires at least one task")

        cases: list[ScenarioCaseResult] = []
        for scenario in self._scenarios:
            for task in tasks:
                result = scenario.simulate(
                    task.messages,
                    model=task.model,
                    provider=task.provider,
                    output_buffer_tokens=task.output_buffer_tokens,
                )
                guarantees = tuple(
                    guarantee(scenario, task, result) for guarantee in self._guarantees
                )
                cases.append(
                    ScenarioCaseResult(
                        scenario=scenario.name,
                        task_id=task.task_id,
                        result=result,

View on GitHub (pinned to 322425c43b)

Solutions

1. Supply at least one ScenarioTask: orchestrator.run([ScenarioTask(messages=[...], model=..., provider=...)]).
2. Verify the fixture path/glob used to build tasks and that files parse (a loader may swallow errors and return []).
3. Guard in test setup: skip the test when the task list is empty instead of calling run().

Example fix

# before
report = orch.run(tasks=load_tasks("./fixtures"))  # empty dir -> ValueError

# after
tasks = load_tasks("./fixtures")
assert tasks, "no scenario tasks loaded"
report = orch.run(tasks)

if not tasks:
    pytest.skip("no scenario tasks available")
report = orch.run(tasks)

When it happens

Trigger: Calling orchestrator.run([]) directly; passing a task list loaded from a directory that was empty (bad glob, wrong fixtures path); tasks filtered out by provider/model predicate before run().

Common situations: Test fixtures directory moved or renamed; CI checkout missing data files; a provider filter (e.g. only openai tasks) removing every task in an anthropic-only environment.

Related errors

• HeadroomSuite requires at least one scenario
• ScenarioOrchestrator requires at least one scenario
• mode must be 'token' or 'cache'
• invalid pipeline config TOML: {0}
• recommendations file not found: {0}

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/3b6877f98ca9459d. Report an issue: GitHub.