{"record":{"id":"da87bae027ccade7","repo":"coleam00/Archon","slug":"run-lookup-failed-not-reaped-err-as-error-m","errorCode":null,"errorMessage":"run lookup failed (NOT reaped): ${(err as Error).message}","messagePattern":"run lookup failed \\(NOT reaped\\): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"packages/core/src/services/cleanup-service.ts","lineNumber":190,"sourceCode":"  const backend = new ContainerBackend({\n    store: isolationEnvDb.createIsolationStore(),\n    config: CLEANUP_PLACEHOLDER_CONTAINER_CONFIG,\n  });\n\n  for (const row of rows) {\n    // FAIL CLOSED on an ambiguous lookup (H3): a DB error is NOT \"no run\" — treating\n    // it as an orphan would destroy a claimable run's container on a transient blip\n    // (violating No-Autonomous-Lifecycle-Mutation). Report + skip, never destroy.\n    //\n    // Same lock the worktree sweeps use. getRunByIsolationEnvId, which this replaced,\n    // took the newest run row BEFORE filtering status, so a newer terminal run could\n    // shadow an older claimable one and the container would be reaped underneath it.\n    let liveRun: Awaited<ReturnType<typeof isolationEnvDb.getLiveRunOwningEnv>>;\n    try {\n      liveRun = await isolationEnvDb.getLiveRunOwningEnv(row.id);\n    } catch (err) {\n      report.errors.push({\n        id: row.id,\n        error: `run lookup failed (NOT reaped): ${(err as Error).message}`,\n      });\n      getLog().warn({ err, envId: row.id }, 'container_env_reap_lookup_failed');\n      continue;\n    }\n    if (liveRun) {\n      report.skipped.push({\n        id: row.id,\n        reason: `run ${liveRun.id.slice(0, 8)} is ${liveRun.status}`,\n      });\n      continue;\n    }\n    if (row.days_since_created < daysStale) {\n      report.skipped.push({\n        id: row.id,\n        reason: `${Math.floor(row.days_since_created)}d old (< ${daysStale}d threshold)`,\n      });\n      continue;","sourceCodeStart":172,"sourceCodeEnd":208,"githubUrl":"https://github.com/coleam00/Archon/blob/0773b9745896ef0612e709c80845a0f7db315b19/packages/core/src/services/cleanup-service.ts#L172-L208","documentation":"A per-environment error recorded during container environment cleanup (cleanupContainerEnvironments) when the lookup of a live run owning the environment (getLiveRunOwningEnv) throws. The environment is deliberately NOT reaped and the loop continues to the next row, because reaping under uncertainty could destroy an environment a live run owns; the failure is logged as container_env_reap_lookup_failed.","triggerScenarios":"During a cleanup sweep over claimable isolation environments, the per-row DB query getLiveRunOwningEnv throws — database unreachable mid-sweep, query timeout, or a transient connection error — for a specific environment id.","commonSituations":"Cleanup service running while the database is being restarted or is under heavy load; connection pool exhausted by concurrent workflows; network blip between the cleanup worker and the DB.","solutions":["Re-run the cleanup report/sweep — the environment was skipped (NOT reaped) and will be reconsidered.","Check the container_env_reap_lookup_failed log entries for the underlying DB error.","Fix database health (connectivity, pool limits, locks) before the next cleanup cycle.","Verify the environment's owning run state manually if it remains uncleaned across multiple sweeps."],"exampleFix":"// before\ncleanup report   # 'run lookup failed (NOT reaped)' for env 7\n// after\npg_isready -h localhost   # restore DB, then\ncleanup report   # env 7 reaped or claimed correctly","handlingStrategy":"retry","validationCode":"// Probe DB reachability before starting a cleanup sweep:\ntry { await db.execute(sql`select 1`); } catch { console.error('DB unreachable; postponing cleanup sweep'); return; }","typeGuard":null,"tryCatchPattern":"try {\n  liveRun = await isolationEnvDb.getLiveRunOwningEnv(envId);\n} catch (err) {\n  getLog().warn({ err, envId }, 'container_env_reap_lookup_failed');\n  // skip: never reap when run ownership is unknown\n  continue;\n}","preventionTips":["Schedule cleanup sweeps away from DB restart/maintenance windows.","Size the connection pool so cleanup queries are not starved by workflows.","Never reap container environments when run ownership cannot be determined.","Re-run failed sweeps promptly so skipped environments do not accumulate."],"tags":["cleanup","database","containers"],"backgroundTag":"database-operation-failed","analyzedSha":"0773b9745896ef0612e709c80845a0f7db315b19","analyzedAt":"2026-09-01T02:28:07.064Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}