{"record":{"id":"5a9120fb742edcf7","repo":"paperclipai/paperclip","slug":"native-replacement-stopped-session-changed","errorCode":"native_replacement_stopped_session_changed","errorMessage":"native_replacement_stopped_session_changed","messagePattern":"native_replacement_stopped_session_changed","errorType":"error_code","errorClass":"Error","httpStatus":null,"severity":"warning","filePath":"server/src/services/native-runtime/native-safe-replacement.ts","lineNumber":361,"sourceCode":"            return receipt.runId === run.id && receipt.statusVersion === task.statusVersion;\n          });\n          if (!ownsBlock) return false;\n        }\n        if (stoppedSession) {\n          const [currentRun] = await tx.select().from(heartbeatRuns).where(and(\n            eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),\n          )).for(\"update\");\n          if (!currentRun || currentRun.status !== \"failed\" || currentRun.runnerInstanceId !== run.runnerInstanceId ||\n              currentRun.nativeSessionId !== run.nativeSessionId || currentRun.processPid || currentRun.processGroupId) return false;\n        }\n        if (task.status === \"blocked\") {\n          // Restore only this failure's unchanged projection. The normal issue\n          // service still enforces dependency readiness and assignee eligibility.\n          await issueService(tx as unknown as Db).update(task.id, { status: \"in_progress\" }, tx);\n        }\n        if (stoppedSession) {\n          // If the last ownership proof changes, roll back the status restoration.\n          if (!stoppedSession.retire()) throw new Error(\"native_replacement_stopped_session_changed\");\n          await appendHeartbeatRunEvent(tx as unknown as Db, {\n            companyId: run.companyId, runId: run.id, agentId: run.agentId,\n            eventType: \"native.stopped_text_turn_verified\", stream: \"system\", level: \"info\",\n            message: \"The previous runner and provider stopped. The interrupted turn had no external actions; any completion bookkeeping has a verified receipt.\",\n            payload: stoppedSession.evidence,\n          });\n        }\n        const successorRunId = randomUUID();\n        const dueAt = new Date(now.getTime() + 30_000);\n        const predecessorContext = { ...record(run.contextSnapshot) };\n        // History comes from the failed source run. Consumed wake fields must\n        // not grant this automatic retry fresh comment/resume authority.\n        for (const key of [\n          \"explicitUserContinuation\", \"wakeCommentId\", \"wakeCommentIds\", \"commentId\",\n          \"commentIds\", \"latestCommentId\", \"resumeIntent\", \"followUpRequested\",\n          \"paperclipWake\", \"paperclipWakeComment\", \"paperclipTaskMarkdown\", \"paperclipTaskMarkdownCompact\",\n        ]) delete predecessorContext[key];\n        const context = {","sourceCodeStart":343,"sourceCodeEnd":379,"githubUrl":"https://github.com/paperclipai/paperclip/blob/3f1d897a7c018d76563a21c6e39c3c9b03933622/server/src/services/native-runtime/native-safe-replacement.ts#L343-L379","documentation":"During reconcileSafeNativeReplacements (the safe native replacement sweep invoked by executionControlSweeps), when a stopped runner session's verified evidence is about to be retired, retire() must succeed confirming the last ownership proof is still valid. If retire() returns false — the ownership proof changed concurrently (run row mutated, session identity changed, a process handle appeared) — the transaction throws this error so the status restoration rolls back and no successor retry run is scheduled on stale ownership evidence.","triggerScenarios":"Concurrent mutation of the heartbeatRuns row between the in-transaction ownership check (status 'failed', matching runnerInstanceId/nativeSessionId, no processPid/processGroupId) and the stoppedSession.retire() call: another sweep, watchdog, or operator action re-claimed, retried, or attached to the run; session evidence was superseded by a newer proof.","commonSituations":"Two executionControlSweeps racing on the same failed native run; an operator manually restarting the runner while the replacement sweep executes; a wake/retry request landing in the same window; optimistic-concurrency CAS inside retire() losing because the DB row's ownership columns changed.","solutions":["No user action needed — this is an expected concurrency guard; the sweep safely rolls back and a later sweep can retry once ownership is stable.","If it recurs persistently, look for another process repeatedly touching heartbeat_runs (restarts, watchdogs) and serialize those against the replacement sweep.","Check logs for a competing recovery action or successorRunId already created for the run; if the replacement already happened, the error is benign noise.","Ensure only one executionControlSweeps instance runs per instance (leader election / single sweeper) to reduce races."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// before scheduling a safe replacement, confirm the run row still proves ownership\nconst [r] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for('update');\nif (!r || r.status !== 'failed' || r.runnerInstanceId !== expectedInstanceId || r.processPid || r.processGroupId) {\n  return; // ownership changed; skip replacement this sweep\n}","typeGuard":null,"tryCatchPattern":"try {\n  await db.transaction(async (tx) => {\n    // ... status restoration + stoppedSession.retire() + successor scheduling\n  });\n} catch (e) {\n  if (e.message === 'native_replacement_stopped_session_changed') {\n    log.info('replacement skipped: ownership proof changed concurrently; will retry next sweep');\n    return; // transaction rolled back; safe to retry later\n  }\n  throw e;\n}","preventionTips":["Ensure only one executionControlSweeps leader runs concurrently (leader election).","Avoid manual runner restarts while recovery sweeps are active; pause the agent first.","Keep retire()'s compare-and-set scoped to the same row version read inside the transaction.","Monitor recurrence; persistent errors indicate another writer is repeatedly mutating heartbeat_runs ownership columns."],"tags":["concurrency","native-runner","race-condition","recovery"],"backgroundTag":"invalid-state-transition","analyzedSha":"3f1d897a7c018d76563a21c6e39c3c9b03933622","analyzedAt":"2026-09-18T08:03:59.046Z","contentChangedAt":"2026-09-18T08:03:59.046Z","schemaVersion":2},"datasetVersion":"2026-09-22T11:17:16.035Z"}