{"record":{"id":"027d32d883c18df2","repo":"paperclipai/paperclip","slug":"provider-checkpoint-failed-terminal","errorCode":"provider_checkpoint_failed_terminal","errorMessage":"provider_checkpoint_failed_terminal","messagePattern":"provider_checkpoint_failed_terminal","errorType":"error_code","errorClass":"NativeProviderTerminalFailure","httpStatus":null,"severity":"error","filePath":"packages/paperclip-runner/src/native-session-runtime.ts","lineNumber":1852,"sourceCode":"        reconcileRecoveryCursor({\n          controlPlane: options.controlPlane,\n          checkpoint: recoveryCheckpoint,\n          runId: input.binding.runId,\n          sourceInstanceId: options.runnerInstanceId,\n          signal,\n        }),\n    });\n    reconciledRecoveryCheckpoint = persistedSession;\n    const providerRecoveryCheckpoint = persistedSession;\n\n    const replacementAllowed =\n      providerRecoveryCheckpoint.providerRecoveryPolicy ===\n      \"allow_replacement_after_resume_failure\";\n    const failedProviderSession =\n      providerRecoveryCheckpoint.terminal?.runTerminalState === \"failed\" &&\n      providerRecoveryCheckpoint.semanticResult === null;\n    if (failedProviderSession && !replacementAllowed) {\n      throw new NativeProviderTerminalFailure(\"provider_checkpoint_failed_terminal\", false);\n    }\n    const recovery = failedProviderSession\n      ? {\n          recovered: false as const,\n          reason: \"provider session ended with a failed terminal\",\n        }\n      : options.backend.recoverSession\n        ? await runAbortableOperationWithin({\n            timeoutMs: recoveryTimeoutMs,\n            timeoutMessage: `native session provider recovery timed out after ${recoveryTimeoutMs}ms`,\n            operation: (signal) =>\n              options.backend.recoverSession!(providerRecoveryCheckpoint, {\n                signal,\n              }),\n            onLateResolution: async (lateRecovery) => {\n              if (lateRecovery.session) {\n                await disposeUnadmittedSession(\n                  lateRecovery.session,","sourceCodeStart":1834,"sourceCodeEnd":1870,"githubUrl":"https://github.com/paperclipai/paperclip/blob/01ad8584922b5d85292b1723cae71fa0d9b07a19/packages/paperclip-runner/src/native-session-runtime.ts#L1834-L1870","documentation":"During provider recovery checkpoint evaluation in the native session runtime, a checkpoint whose terminal.runTerminalState is 'failed' with a null semanticResult marks the provider session as terminally failed. If the checkpoint's providerRecoveryPolicy is not 'allow_replacement_after_resume_failure', the runtime throws NativeProviderTerminalFailure with code provider_checkpoint_failed_terminal (retryable=false), refusing to resume a dead provider session. When replacement IS allowed, it instead returns recovered:false with a reason instead of throwing.","triggerScenarios":"Resuming a run whose provider session (e.g. a CLI agent process) ended in a failed terminal state and produced no semantic result, while the recovery policy for that provider does not permit replacement after resume failure — e.g. resume-after-crash of a session that exited non-zero with no usable output.","commonSituations":"Provider CLI crashed or was OOM-killed mid-run; provider exited with a fatal error before emitting a result; resuming old sessions after a provider version upgrade that changed exit semantics; policy not configured to allow replacement, so the runtime treats the failure as terminal instead of silently substituting a new session.","solutions":["Configure the provider recovery policy to 'allow_replacement_after_resume_failure' if starting a replacement provider session is acceptable for this workload","Investigate why the provider session failed (check provider logs/exit state in the checkpoint's terminal.runTerminalState) before resuming","Rerun the task from scratch rather than resuming, since recovery is blocked for failed provider sessions","If failures cluster after a provider upgrade, pin or roll back the provider version and re-checkpoint sessions","Handle NativeProviderTerminalFailure as non-retryable in the calling loop — retrying resume will throw again"],"exampleFix":"// before (policy blocks replacement)\nproviderRecoveryPolicy: 'never_replace';\n// after\nproviderRecoveryPolicy: 'allow_replacement_after_resume_failure';\n// resume then returns { recovered: false, reason: 'provider session ended with a failed terminal' } instead of throwing","handlingStrategy":"try-catch","validationCode":"const cp = await loadProviderRecoveryCheckpoint(runId); const terminal = cp.terminal?.runTerminalState === 'failed' && cp.semanticResult === null; if (terminal && cp.providerRecoveryPolicy !== 'allow_replacement_after_resume_failure') throw new Error('provider session terminally failed; replacement not allowed');","typeGuard":"function isProviderTerminalFailure(e: unknown) { return e instanceof NativeProviderTerminalFailure && e.message === 'provider_checkpoint_failed_terminal'; }","tryCatchPattern":"try { await resumeRun(runId); } catch (e) { if (isProviderTerminalFailure(e)) { return restartRunFromScratch(runId); } throw e; }","preventionTips":["Set providerRecoveryPolicy to 'allow_recovery_after_resume_failure' when replacement sessions are acceptable","Capture provider logs and terminal state to diagnose why the session failed before resuming","Pin provider versions so upgrades do not strand in-flight sessions in failed terminals","Treat this error as non-retryable — do not loop on resume attempts"],"tags":["recovery","provider-session","terminal-failure","runner","non-retryable"],"backgroundTag":"invalid-state-transition","analyzedSha":"01ad8584922b5d85292b1723cae71fa0d9b07a19","analyzedAt":"2026-09-10T03:14:50.855Z","contentChangedAt":"2026-09-10T03:14:50.855Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}