paperclipai/paperclip · warning · Error

native_replacement_stopped_session_changed

native_replacement_stopped_session_changed

Error message

native_replacement_stopped_session_changed

What it means

During reconcileSafeNativeReplacements (the safe native replacement sweep invoked by executionControlSweeps), when a stopped runner session's verified evidence is about to be retired, retire() must succeed confirming the last ownership proof is still valid. If retire() returns false — the ownership proof changed concurrently (run row mutated, session identity changed, a process handle appeared) — the transaction throws this error so the status restoration rolls back and no successor retry run is scheduled on stale ownership evidence.

Solutions

1. No user action needed — this is an expected concurrency guard; the sweep safely rolls back and a later sweep can retry once ownership is stable.
2. If it recurs persistently, look for another process repeatedly touching heartbeat_runs (restarts, watchdogs) and serialize those against the replacement sweep.
3. Check logs for a competing recovery action or successorRunId already created for the run; if the replacement already happened, the error is benign noise.
4. Ensure only one executionControlSweeps instance runs per instance (leader election / single sweeper) to reduce races.

// before scheduling a safe replacement, confirm the run row still proves ownership
const [r] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for('update');
if (!r || r.status !== 'failed' || r.runnerInstanceId !== expectedInstanceId || r.processPid || r.processGroupId) {
  return; // ownership changed; skip replacement this sweep
}

try {
  await db.transaction(async (tx) => {
    // ... status restoration + stoppedSession.retire() + successor scheduling
  });
} catch (e) {
  if (e.message === 'native_replacement_stopped_session_changed') {
    log.info('replacement skipped: ownership proof changed concurrently; will retry next sweep');
    return; // transaction rolled back; safe to retry later
  }
  throw e;
}

When it happens

Trigger: Concurrent mutation of the heartbeatRuns row between the in-transaction ownership check (status 'failed', matching runnerInstanceId/nativeSessionId, no processPid/processGroupId) and the stoppedSession.retire() call: another sweep, watchdog, or operator action re-claimed, retried, or attached to the run; session evidence was superseded by a newer proof.

Common situations: Two executionControlSweeps racing on the same failed native run; an operator manually restarting the runner while the replacement sweep executes; a wake/retry request landing in the same window; optimistic-concurrency CAS inside retire() losing because the DB row's ownership columns changed.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Bridge envelope changed while reading.
• Capability live turn admission was abandoned during teardown
• Capability live turn start completed after admission…
• CreateOS lease cleanup is already in progress.
• Photon send identity changed

AI-assisted analysis of paperclipai/paperclip@3f1d897a7c (2026-09-18). Data as JSON: /api/errors/5a9120fb742edcf7. Report an issue: GitHub.

Appendix: source

Thrown at server/src/services/native-runtime/native-safe-replacement.ts:361

            return receipt.runId === run.id && receipt.statusVersion === task.statusVersion;
          });
          if (!ownsBlock) return false;
        }
        if (stoppedSession) {
          const [currentRun] = await tx.select().from(heartbeatRuns).where(and(
            eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
          )).for("update");
          if (!currentRun || currentRun.status !== "failed" || currentRun.runnerInstanceId !== run.runnerInstanceId ||
              currentRun.nativeSessionId !== run.nativeSessionId || currentRun.processPid || currentRun.processGroupId) return false;
        }
        if (task.status === "blocked") {
          // Restore only this failure's unchanged projection. The normal issue
          // service still enforces dependency readiness and assignee eligibility.
          await issueService(tx as unknown as Db).update(task.id, { status: "in_progress" }, tx);
        }
        if (stoppedSession) {
          // If the last ownership proof changes, roll back the status restoration.
          if (!stoppedSession.retire()) throw new Error("native_replacement_stopped_session_changed");
          await appendHeartbeatRunEvent(tx as unknown as Db, {
            companyId: run.companyId, runId: run.id, agentId: run.agentId,
            eventType: "native.stopped_text_turn_verified", stream: "system", level: "info",
            message: "The previous runner and provider stopped. The interrupted turn had no external actions; any completion bookkeeping has a verified receipt.",
            payload: stoppedSession.evidence,
          });
        }
        const successorRunId = randomUUID();
        const dueAt = new Date(now.getTime() + 30_000);
        const predecessorContext = { ...record(run.contextSnapshot) };
        // History comes from the failed source run. Consumed wake fields must
        // not grant this automatic retry fresh comment/resume authority.
        for (const key of [
          "explicitUserContinuation", "wakeCommentId", "wakeCommentIds", "commentId",
          "commentIds", "latestCommentId", "resumeIntent", "followUpRequested",
          "paperclipWake", "paperclipWakeComment", "paperclipTaskMarkdown", "paperclipTaskMarkdownCompact",
        ]) delete predecessorContext[key];
        const context = {

View on GitHub (pinned to 3f1d897a7c)