{"record":{"id":"8666f3fa6e9d2d13","repo":"paperclipai/paperclip","slug":"createos-process-output-connection-failed","errorCode":null,"errorMessage":"CreateOS process output connection failed.","messagePattern":"CreateOS process output connection failed\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/plugins/sandbox-providers/createos/src/execute.ts","lineNumber":145,"sourceCode":"    processId = identifier(created.process_id);\n    // No interactive stdin for ordinary commands; any supplied input comes\n    // from the staged file, avoiding a write-vs-fast-exit race.\n    try { await client.json(`${base}/${processId}/stdin/close`, \"POST\", undefined, signal); }\n    catch (error) {\n      if (!(error instanceof CreateosApiError && error.status === 409)) throw error;\n    }\n    let reconnects = 0;\n    for (;;) {\n      signal.throwIfAborted();\n      let response: Response;\n      try {\n        response = await client.request(`${base}/${processId}/connect?after=${cursor}`, { signal });\n      } catch (error) {\n        if (error instanceof CreateosApiError && error.status === 410) {\n          throw new Error(\"CreateOS process output was evicted before it could be read.\");\n        }\n        if (signal.aborted || (error instanceof CreateosApiError && error.status < 500 && error.status !== 429)) throw error;\n        if (++reconnects > 3) throw new Error(\"CreateOS process output connection failed.\");\n        await delay(250, undefined, { signal });\n        continue;\n      }\n      try {\n        for await (const event of events(response)) {\n          if (event.type === \"heartbeat\") continue;\n          if (event.type === \"data\") {\n            const seq = event.seq;\n            if (typeof seq !== \"number\" || !Number.isSafeInteger(seq) || seq < 1) throw new Error(\"CreateOS process sequence is invalid.\");\n            if (seq <= cursor) continue;\n            if (seq !== cursor + 1) throw new Error(\"CreateOS process output has a sequence gap.\");\n            if ((event.stream !== \"stdout\" && event.stream !== \"stderr\") ||\n                typeof event.data_base64 !== \"string\" ||\n                !/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(event.data_base64)) {\n              throw new Error(\"CreateOS process output is invalid.\");\n            }\n            output.write(event.stream, Buffer.from(event.data_base64, \"base64\"));\n            cursor = seq;","sourceCodeStart":127,"sourceCodeEnd":163,"githubUrl":"https://github.com/paperclipai/paperclip/blob/3f1d897a7c018d76563a21c6e39c3c9b03933622/packages/plugins/sandbox-providers/createos/src/execute.ts#L127-L163","documentation":"The plugin failed to (re)establish the /processes/:id/connect connection more than 3 times (only 5xx/429 errors are retried, with 250 ms backoff). This means the output stream endpoint is persistently unreachable or erroring, so process output cannot be read.","triggerScenarios":"Four consecutive connect attempts fail with 5xx or 429 (or a network error classified as retryable), e.g. the CreateOS host is down, DNS fails, TLS fails, or the server is rate-limiting /connect beyond the retry budget.","commonSituations":"CreateOS service restart or outage during a command; aggressive rate limiting after many concurrent executes; firewall/DNS misconfiguration in the sandbox base URL; load balancer returning 503s for the connect route.","solutions":["Retry the whole execute call after verifying the CreateOS service is healthy (/health or equivalent).","Check rate limiting: 429 responses exhaust the 3-retry budget quickly; reduce concurrency or add backoff at the caller level.","Verify the configured CreateOS base URL, DNS, and network reachability to the sandbox host.","Inspect server logs for 5xx on the /connect route during the failure window.","Note the process tree is terminated by cleanup on this error; a rerun is safe."],"exampleFix":"// before\nawait client.request(`${base}/${processId}/connect?after=${cursor}`); // fails, no budget left\n// after (caller-side)\nfor (let i = 0; i < 3; i++) { try { return await execute(client, params, signal); } catch (e) { if (!/connection failed/.test(e.message)) throw e; await sleep(2 ** i * 250); } }","handlingStrategy":"retry","validationCode":"// Pre-flight reachability check\nconst probe = await fetch(`${baseUrl}/sandboxes/${leaseId}/processes`, { method: \"HEAD\" }).catch(() => null);\nif (!probe) throw new Error(\"CreateOS host unreachable before execute\");","typeGuard":null,"tryCatchPattern":"try { await execute(client, params, signal); }\ncatch (e) {\n  if (e instanceof Error && e.message === \"CreateOS process output connection failed.\") {\n    await sleep(1000); // service-level backoff, then re-run execute (cleanup already killed the tree)\n  } else throw e;\n}","preventionTips":["Check CreateOS service health before batch runs.","Respect 429 rate limits; throttle concurrent executes.","Verify base URL, DNS, and TLS configuration.","Monitor 5xx rates on the /connect route."],"tags":["network","connection","retry-exhausted"],"backgroundTag":"http-request-failed","analyzedSha":"3f1d897a7c018d76563a21c6e39c3c9b03933622","analyzedAt":"2026-09-18T08:03:59.046Z","contentChangedAt":"2026-09-18T08:03:59.046Z","schemaVersion":2},"datasetVersion":"2026-09-22T01:17:13.364Z"}