{"record":{"id":"b7940229f1ab4da0","repo":"paperclipai/paperclip","slug":"paperclip-sandbox-callback-bridge-kept-queued-re","errorCode":null,"errorMessage":"[paperclip] sandbox callback bridge kept queued request ${requestId} after every recovery 503 write failed: ${lastWriteError}","messagePattern":"\\[paperclip\\] sandbox callback bridge kept queued request (.+?) after every recovery 503 write failed: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"packages/adapter-utils/src/sandbox-callback-bridge.ts","lineNumber":1285,"sourceCode":"          wrote503 = true;\n          break;\n        } catch (error) {\n          lastWriteError = error instanceof Error ? error.message : String(error);\n          console.warn(\n            `[paperclip] sandbox callback bridge failed to write recovery 503 for ${requestId} (attempt ${attempt}/${MAX_BACKSTOP_WRITE_ATTEMPTS}): ${lastWriteError}`,\n          );\n          if (attempt < MAX_BACKSTOP_WRITE_ATTEMPTS) {\n            await new Promise((resolve) => setTimeout(resolve, BACKSTOP_WRITE_RETRY_MS));\n          }\n        }\n      }\n      if (wrote503) {\n        // The 503 landed. Remove the request file, so the poll loop does not\n        // re-process it.\n        await input.client.remove(requestPath).catch(() => undefined);\n      } else {\n        // Every 503 write failed. Keep the request file for a later recovery pass.\n        console.warn(\n          `[paperclip] sandbox callback bridge kept queued request ${requestId} after every recovery 503 write failed: ${lastWriteError}`,\n        );\n      }\n    }\n  };\n\n  // Surface a bridge-worker failure through the run trace, not only stdout. A\n  // failed span under `input.runtimeSpan` records the error against the live run\n  // span, so the run and the orchestrator see the hang. When no `runtimeSpan`\n  // runner is wired (no injected tracer), the helper still writes a warn line,\n  // so the failure is never silent.\n  const surfaceRunError = async (error: Error) => {\n    if (input.runtimeSpan) {\n      try {\n        await input.runtimeSpan(CALLBACK_BRIDGE_WORKER_FAILED_SPAN, async () => {\n          throw error;\n        });\n      } catch {","sourceCodeStart":1267,"sourceCodeEnd":1303,"githubUrl":"https://github.com/paperclipai/paperclip/blob/a7e689b3c35347b529cb9f54c9b9a8575a3dcab6/packages/adapter-utils/src/sandbox-callback-bridge.ts#L1267-L1303","documentation":"In the sandbox callback bridge's recovery path (failPendingRequests), a queued request that was never claimed must be answered with a terminal 503 response file. The write is retried up to MAX_BACKSTOP_WRITE_ATTEMPTS times, each bounded by the per-iteration timeout. If every attempt fails (lastWriteError), the bridge deliberately keeps the request .json file so a later recovery pass or caller retry can still deliver the 503 — dropping it would strand the sandbox caller until its own deadline. The warning names the retained requestId.","triggerScenarios":"The sandbox file channel becomes unresponsive (frozen/paused sandbox, dead transport, host I/O failure) while callback requests sit in the requests directory — reads/list succeeded enough to enumerate the request, but every 503 write times out or errors.","commonSituations":"Sandbox container paused by the host; gVisor/FUSE-backed channel stalling; long GC pauses; storage outage during a run with in-flight callbacks.","solutions":["Check sandbox health: is the sandbox process/container alive and the shared directory writable? Restart or resume it.","Let a later recovery pass handle the retained request — re-running the agent turn re-creates the bridge and re-enumerates the queue.","Inspect run logs for the sibling warnings (failed read / failed 503 write per attempt) to confirm the channel, not the code, is the failure point.","If the sandbox is unrecoverable, stop the run: the sandbox caller is designed to hit its own deadline and surface a timeout."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Smoke-test the channel before relying on recovery writes\ntry {\n  await withTimeout(input.client.listJsonFiles(directories.requestsDir), 2_000, \"channel probe\");\n} catch {\n  // channel unresponsive: skip recovery sweep, let a later pass deliver 503s\n}","typeGuard":"const hasQueuedRequests = (files: string[]): boolean => files.some((f) => f.endsWith(\".json\"));","tryCatchPattern":"for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {\n  try { await writeBridgeResponse(...); wrote = true; break; }\n  catch (e) { lastWriteError = e; await delay(BACKOFF_MS); }\n}\nif (!wrote) {\n  // keep the request file — never drop it; a later pass or caller retry handles it\n  console.warn(`kept queued request ${requestId}: ${lastWriteError}`);\n}","preventionTips":["Keep the sandbox (and its shared storage) alive and responsive while callbacks are in flight; pause/kill only at turn boundaries.","Bound every channel op with the per-iteration timeout so a dead channel fails fast instead of hanging the sweep.","Monitor for this warn: it means degraded mode — queued callers are waiting on a later recovery pass.","Never 'clean up' leftover request .json files manually during a live run; they are retained deliberately."],"tags":["sandbox","callback-bridge","recovery","queue","i-o-failure"],"backgroundTag":"sandbox-callback-write-failed","analyzedSha":"a7e689b3c35347b529cb9f54c9b9a8575a3dcab6","analyzedAt":"2026-08-18T22:49:45.177Z","schemaVersion":2},"datasetVersion":"2026-08-22T09:17:25.309Z"}