{"record":{"id":"8f07eb47303ca80a","repo":"coleam00/Archon","slug":"executor-backstop-triggered","errorCode":"executor.backstop_triggered","errorMessage":"Workflow exited without finalizing — see logs","messagePattern":"Workflow exited without finalizing — see logs","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/workflows/src/executor.ts","lineNumber":3237,"sourceCode":"\n    // Return failure result instead of re-throwing\n    return { success: false, workflowRunId: workflowRun.id, error: err.message };\n  } finally {\n    // Release the keep-awake request FIRST — before the backstop DB calls that\n    // may throw — so it always pairs with the acquire above this try, on every\n    // exit path (success, thrown error, or backstop failure).\n    keepAwake.release();\n    // Defensive backstop: if the workflow run is still 'running' after all\n    // normal and exceptional code paths, flip it to 'failed' to prevent zombie\n    // accumulation. Guards against any future code path that exits without\n    // calling failWorkflowRun (e.g. a generator cleanup that exits without\n    // throwing). Only fires when the process stays alive long enough to run\n    // this finally — see #1561 for the originating zombie-state incident.\n    if (workflowRun && !terminalStatusWriteFailed) {\n      const runId = workflowRun.id;\n      const backstopStatus = await deps.store.getWorkflowRunStatus(runId).catch(() => null);\n      if (backstopStatus === 'running') {\n        getLog().warn({ workflowRunId: runId }, 'executor.backstop_triggered');\n        await requireTerminalStatusWrite(\n          deps.store.failWorkflowRun(runId, 'Workflow exited without finalizing — see logs'),\n          { workflowRunId: runId, site: 'executor.backstop_fail_failed' }\n        );\n      }\n    }\n  }\n}\n","sourceCodeStart":3219,"sourceCodeEnd":3246,"githubUrl":"https://github.com/coleam00/Archon/blob/0773b9745896ef0612e709c80845a0f7db315b19/packages/workflows/src/executor.ts#L3219-L3246","documentation":"A safety backstop in the executor's finally block: if the workflow function exits (returns or throws) while the run is still recorded as 'running' and no terminal status write already failed, the run would remain a zombie. The backstop marks it failed with this message so terminal state always lands. It exists because of the zombie-state incident #1561; the message directs you to logs for the real cause.","triggerScenarios":"The executor's finally block runs with workflowRun set, terminalStatusWriteFailed false, and store.getWorkflowRunStatus(runId) === 'running' — i.e. the workflow body returned early or threw an error that bypassed the normal finalize path (executor.ts:3237), and the process stayed alive long enough for the finally to execute.","commonSituations":"An uncaught exception in a node handler outside normal error handling; a code change adding an early return before finalization; an abort/OOM mid-node skipping completion; custom node types that swallow control flow.","solutions":["Check the executor logs for the workflowRunId to find the exception or early return that bypassed finalization.","Fix the root cause in the workflow/node code that threw or returned without reaching finalize, then re-run the workflow.","Reconcile the zombie run via CLI (`archon workflow resume` or cancel) if a status write raced or was lost.","Add tests covering throw and early-return exit paths so finalize is always reached."],"exampleFix":"// before: early return that skips finalize\nif (!nodes.length) return;\n// after: route through the executor's error/finalize path\nif (!nodes.length) {\n  throw new Error('Workflow has no nodes'); // executor terminalizes the run\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  await executeWorkflow(/* ... */);\n} catch (err) {\n  // guarantee terminalization even on unexpected throws\n  await store.failWorkflowRun(runId, `Unhandled executor error: ${err}`);\n  throw err;\n}","preventionTips":["Never add early returns inside the workflow execution function that bypass the finalize path.","Wrap custom node handlers so unexpected throws propagate through the executor's normal failure handling.","Monitor logs for executor.backstop_triggered — each occurrence is a finalize-path bug to fix, not just a run to restart.","Add regression tests for throw and early-return exit paths in custom node types."],"tags":["workflow","executor","lifecycle","zombie-state"],"backgroundTag":"zombie-run-state","analyzedSha":"0773b9745896ef0612e709c80845a0f7db315b19","analyzedAt":"2026-09-01T02:28:07.064Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}