{"record":{"id":"335d00895c33b5c1","repo":"bytedance/deer-flow","slug":"fan-out-failed-for-delivery-x-github-delivery-r","errorCode":null,"errorMessage":"fan-out failed for delivery {x_github_delivery!r}: {exc!r}","messagePattern":"fan-out failed for delivery (.+?): (.+?)","errorType":"http","errorClass":"HTTPException","httpStatus":503,"severity":"error","filePath":"backend/app/gateway/routers/github_webhooks.py","lineNumber":365,"sourceCode":"            # failed so that same manual click, REST call, or recovery\n            # script actually finds and recovers it. The startup-time\n            # ``is_route_enabled`` check still covers fail-closed\n            # *configuration* errors.\n            try:\n                dispatch_result = await fanout_event(\n                    service.bus,\n                    x_github_event,\n                    x_github_delivery,\n                    payload,\n                    operator_default_mention_login=operator_default_mention_login,\n                )\n            except Exception as exc:  # noqa: BLE001 — re-raised as 503 below\n                logger.exception(\n                    \"github_webhook: fanout failed (delivery=%s event=%s) — returning 503 (recoverable via manual/API redelivery)\",\n                    x_github_delivery,\n                    x_github_event,\n                )\n                raise HTTPException(\n                    status_code=503,\n                    detail=f\"fan-out failed for delivery {x_github_delivery!r}: {exc!r}\",\n                ) from exc\n    else:\n        logger.info(\n            \"github_webhook delivery=%s | unhandled event=%s action=%s repo=%s\",\n            x_github_delivery,\n            x_github_event,\n            payload.get(\"action\"),\n            (payload.get(\"repository\") or {}).get(\"full_name\"),\n        )\n        handled = False\n        dispatch_result = None\n\n    return {\n        \"ok\": True,\n        \"event\": x_github_event,\n        \"delivery\": x_github_delivery,","sourceCodeStart":347,"sourceCodeEnd":383,"githubUrl":"https://github.com/bytedance/deer-flow/blob/1dd6ba1acb03700589994b0366c5d1c7d05e2eff/backend/app/gateway/routers/github_webhooks.py#L347-L383","documentation":"Raised by the GitHub webhook receiver with status 503 when publishing the verified event onto the channel bus (fan-out to custom agents via the ChannelManager) raises any exception. 503 signals a recoverable server-side failure so GitHub will retry the delivery; the message embeds the delivery id and the causing exception for correlation with the logged traceback.","triggerScenarios":"Channel bus backend down (Redis/stream store unavailable); a channel adapter raising on a malformed repo/action combination; agent-routing state missing after a partial restart; serialization errors on unusual payload shapes.","commonSituations":"Channel/IM infrastructure (the bus the ChannelManager consumes) not running when webhooks arrive; transient DB/stream outage; version skew during rolling deploys where the event publisher and channel service disagree on schema.","solutions":["Check Gateway logs for the 'github_webhook: fanout failed' exception traceback and fix the underlying bus/channel error it names.","Verify the channel bus service is up (same stack: make docker-logs / service health endpoints).","After fixing, redeliver the failed event from GitHub's webhook delivery log or via the API — 503 tells GitHub to retry automatically within its schedule."],"exampleFix":"# before: bus down, delivery lost from view\n# webhook logs: fanout failed ... 503\n\n# after\nmake docker-logs SERVICE=gateway  # read traceback, e.g. stream store connection refused\nmake up                          # restore the bus, GitHub auto-retries (or Redeliver in webhook UI)","handlingStrategy":"retry","validationCode":"# preflight the dependency the fan-out needs\n curl -fsS http://gateway:8001/health || echo 'gateway unhealthy — fix before expecting webhook fan-out'","typeGuard":null,"tryCatchPattern":"try { await postWebhook(payload); } catch (e) { if (e.status === 503) { await backoffRetry(3); } throw e; }","preventionTips":["Keep the channel bus running whenever webhooks are enabled","Monitor Gateway logs for 'fanout failed' tracebacks","Use GitHub redelivery after fixing the root cause — 503 is retryable"],"tags":["http-503","webhooks","fan-out","message-bus","retryable"],"backgroundTag":null,"analyzedSha":"1dd6ba1acb03700589994b0366c5d1c7d05e2eff","analyzedAt":"2026-08-14T21:20:34.804Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}