headroomlabs-ai/headroom · warning · SystemExit

Evaluation interrupted.

Error message

Evaluation interrupted.

What it means

The memory eval runner wraps asyncio.run(evaluator.run()) in a try/except KeyboardInterrupt. If the user interrupts the eval (Ctrl-C / SIGINT) before it finishes, the command prints 'Evaluation interrupted.' and exits with code 1 via SystemExit. This is a controlled cancellation path — there is no partial-result checkpoint; results are only printed/saved after a full run.

Source

Thrown at headroom/cli/evals.py:583

  LLM Judge:        {judge_info}
  Parallelism:      {parallel} workers
  Debug:            {"ENABLED" if debug else "DISABLED"}

Running evaluation...
""")

    # Run evaluation
    evaluator = LoCoMoEvaluator(
        answer_fn=answer_fn,
        llm_judge_fn=llm_judge_fn,
        config=eval_config,
    )

    try:
        result = asyncio.run(evaluator.run())
    except KeyboardInterrupt:
        click.echo("\nEvaluation interrupted.")
        raise SystemExit(1) from None

    # Print results
    click.echo(result.summary())

    # Save results if output path specified
    if output:
        result.save(output)
        click.echo(f"\nResults saved to: {output}")


def _run_memory_eval_v2(
    *,
    n_conversations: int | None,
    categories: str | None,
    include_adversarial: bool,
    f1_threshold: float,
    save_model: str,
    answer_model: str,

View on GitHub (pinned to 322425c43b)

Solutions

1. Re-run the eval with a smaller --n-conversations or a restricted --categories set so it completes before you need the terminal
2. If you interrupted by accident, just re-invoke the command — no state is corrupted, the abort is clean
3. Run long evals under nohup/tmux so stray Ctrl-C does not reach them
4. Use --output to save results and check whether the file exists before assuming a completed run

try:
    subprocess.run(["headroom", "evals", "memory", "--output", "r.json"], check=True)
except subprocess.CalledProcessError as e:
    if e.returncode == 1 and not Path("r.json").exists():
        print("eval did not complete (interrupted) — rerunning is safe")

When it happens

Trigger: Pressing Ctrl-C (or sending SIGINT to the process group) while `headroom evals memory` is running conversations, judging, or waiting on model API calls.

Common situations: Long evals interrupted mid-run; terminal timeouts sending INT; users expecting partial results after aborting.

Related errors

• Error: Memory eval dependencies not installed.
• Error: litellm required for --answer-model. Run: pip install
• Error: Memory eval V2 dependencies not installed.
• Shutting down...
• Error: {e}

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/b15d64b3f1b77ad2. Report an issue: GitHub.