{"record":{"id":"b15d64b3f1b77ad2","repo":"headroomlabs-ai/headroom","slug":"evaluation-interrupted","errorCode":null,"errorMessage":"Evaluation interrupted.","messagePattern":"Evaluation interrupted\\.","errorType":"console","errorClass":"SystemExit","httpStatus":null,"severity":"warning","filePath":"headroom/cli/evals.py","lineNumber":583,"sourceCode":"  LLM Judge:        {judge_info}\n  Parallelism:      {parallel} workers\n  Debug:            {\"ENABLED\" if debug else \"DISABLED\"}\n\nRunning evaluation...\n\"\"\")\n\n    # Run evaluation\n    evaluator = LoCoMoEvaluator(\n        answer_fn=answer_fn,\n        llm_judge_fn=llm_judge_fn,\n        config=eval_config,\n    )\n\n    try:\n        result = asyncio.run(evaluator.run())\n    except KeyboardInterrupt:\n        click.echo(\"\\nEvaluation interrupted.\")\n        raise SystemExit(1) from None\n\n    # Print results\n    click.echo(result.summary())\n\n    # Save results if output path specified\n    if output:\n        result.save(output)\n        click.echo(f\"\\nResults saved to: {output}\")\n\n\ndef _run_memory_eval_v2(\n    *,\n    n_conversations: int | None,\n    categories: str | None,\n    include_adversarial: bool,\n    f1_threshold: float,\n    save_model: str,\n    answer_model: str,","sourceCodeStart":565,"sourceCodeEnd":601,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/cli/evals.py#L565-L601","documentation":"The memory eval runner wraps asyncio.run(evaluator.run()) in a try/except KeyboardInterrupt. If the user interrupts the eval (Ctrl-C / SIGINT) before it finishes, the command prints 'Evaluation interrupted.' and exits with code 1 via SystemExit. This is a controlled cancellation path — there is no partial-result checkpoint; results are only printed/saved after a full run.","triggerScenarios":"Pressing Ctrl-C (or sending SIGINT to the process group) while `headroom evals memory` is running conversations, judging, or waiting on model API calls.","commonSituations":"Long evals interrupted mid-run; terminal timeouts sending INT; users expecting partial results after aborting.","solutions":["Re-run the eval with a smaller --n-conversations or a restricted --categories set so it completes before you need the terminal","If you interrupted by accident, just re-invoke the command — no state is corrupted, the abort is clean","Run long evals under nohup/tmux so stray Ctrl-C does not reach them","Use --output to save results and check whether the file exists before assuming a completed run"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    subprocess.run([\"headroom\", \"evals\", \"memory\", \"--output\", \"r.json\"], check=True)\nexcept subprocess.CalledProcessError as e:\n    if e.returncode == 1 and not Path(\"r.json\").exists():\n        print(\"eval did not complete (interrupted) — rerunning is safe\")","preventionTips":["Size --n-conversations to the time you have; interrupted runs produce nothing","Run long evals in tmux/nohup so terminal Ctrl-C cannot reach them","Treat absence of the --output file as 'run incomplete', never as partial results"],"tags":["cli","evals","keyboard-interrupt","cancellation"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}