{"record":{"id":"b7940229f1ab4da0","repo":"paperclipai/paperclip","slug":"paperclip-sandbox-callback-bridge-kept-queued-re","errorCode":null,"errorMessage":"[paperclip] sandbox callback bridge kept queued request ${requestId} after every recovery 503 write failed: ${lastWriteError}","messagePattern":"\\[paperclip\\] sandbox callback bridge kept queued request (.+?) after every recovery 503 write failed: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"packages/adapter-utils/src/sandbox-callback-bridge.ts","lineNumber":1403,"sourceCode":"          await withTimeout(\n            writeBridgeResponse(input.client, requestPath, responsePath, {\n              id: responseId,\n              status: 503,\n              headers: { \"content-type\": \"application/json\" },\n              body: JSON.stringify({ error: message }),\n              completedAt: new Date().toISOString(),\n            }, {\n              requireRequestPath: false,\n            }),\n            iterationTimeoutMs,\n            `Sandbox callback bridge write 503 for ${requestId}`,\n          );\n          wrote503 = true;\n          break;\n        } catch (error) {\n          lastWriteError = error instanceof Error ? error.message : String(error);\n          console.warn(\n            `[paperclip] sandbox callback bridge failed to write recovery 503 for ${requestId} (attempt ${attempt}/${MAX_BACKSTOP_WRITE_ATTEMPTS}): ${lastWriteError}`,\n          );\n          if (attempt < MAX_BACKSTOP_WRITE_ATTEMPTS) {\n            await new Promise((resolve) => setTimeout(resolve, BACKSTOP_WRITE_RETRY_MS));\n          }\n        }\n      }\n      if (wrote503) {\n        // The 503 landed. Remove the request file, so the poll loop does not\n        // re-process it.\n        await input.client.remove(requestPath).catch(() => undefined);\n      } else {\n        // Every 503 write failed. Keep the request file for a later recovery pass.\n        console.warn(\n          `[paperclip] sandbox callback bridge kept queued request ${requestId} after every recovery 503 write failed: ${lastWriteError}`,\n        );\n      }\n    }\n  };","sourceCodeStart":1385,"sourceCodeEnd":1421,"githubUrl":"https://github.com/paperclipai/paperclip/blob/3f1d897a7c018d76563a21c6e39c3c9b03933622/packages/adapter-utils/src/sandbox-callback-bridge.ts#L1385-L1421","documentation":"In the sandbox callback bridge's recovery path (failPendingRequests), a queued request that was never claimed must be answered with a terminal 503 response file. The write is retried up to MAX_BACKSTOP_WRITE_ATTEMPTS times, each bounded by the per-iteration timeout. If every attempt fails (lastWriteError), the bridge deliberately keeps the request .json file so a later recovery pass or caller retry can still deliver the 503 — dropping it would strand the sandbox caller until its own deadline. The warning names the retained requestId.","triggerScenarios":"The sandbox file channel becomes unresponsive (frozen/paused sandbox, dead transport, host I/O failure) while callback requests sit in the requests directory — reads/list succeeded enough to enumerate the request, but every 503 write times out or errors.","commonSituations":"Sandbox container paused by the host; gVisor/FUSE-backed channel stalling; long GC pauses; storage outage during a run with in-flight callbacks.","solutions":["Check sandbox health: is the sandbox process/container alive and the shared directory writable? Restart or resume it.","Let a later recovery pass handle the retained request — re-running the agent turn re-creates the bridge and re-enumerates the queue.","Inspect run logs for the sibling warnings (failed read / failed 503 write per attempt) to confirm the channel, not the code, is the failure point.","If the sandbox is unrecoverable, stop the run: the sandbox caller is designed to hit its own deadline and surface a timeout."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Smoke-test the channel before relying on recovery writes\ntry {\n  await withTimeout(input.client.listJsonFiles(directories.requestsDir), 2_000, \"channel probe\");\n} catch {\n  // channel unresponsive: skip recovery sweep, let a later pass deliver 503s\n}","typeGuard":"const hasQueuedRequests = (files: string[]): boolean => files.some((f) => f.endsWith(\".json\"));","tryCatchPattern":"for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {\n  try { await writeBridgeResponse(...); wrote = true; break; }\n  catch (e) { lastWriteError = e; await delay(BACKOFF_MS); }\n}\nif (!wrote) {\n  // keep the request file — never drop it; a later pass or caller retry handles it\n  console.warn(`kept queued request ${requestId}: ${lastWriteError}`);\n}","preventionTips":["Keep the sandbox (and its shared storage) alive and responsive while callbacks are in flight; pause/kill only at turn boundaries.","Bound every channel op with the per-iteration timeout so a dead channel fails fast instead of hanging the sweep.","Monitor for this warn: it means degraded mode — queued callers are waiting on a later recovery pass.","Never 'clean up' leftover request .json files manually during a live run; they are retained deliberately."],"tags":["sandbox","callback-bridge","recovery","queue","i-o-failure"],"backgroundTag":"sandbox-callback-write-failed","analyzedSha":"3f1d897a7c018d76563a21c6e39c3c9b03933622","analyzedAt":"2026-08-18T22:49:45.177Z","contentChangedAt":"2026-08-18T22:49:45.177Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}