{"record":{"id":"59233abb8d9a451d","repo":"risingwavelabs/risingwave","slug":"failed-to-choose-a-compute-node-for-fast-insert","errorCode":null,"errorMessage":"Failed to choose a compute node for fast insert","messagePattern":"Failed to choose a compute node for fast insert","errorType":"http","errorClass":null,"httpStatus":503,"severity":"error","filePath":"src/frontend/src/webhook/mod.rs","lineNumber":281,"sourceCode":"                        anyhow!(\"Table `{}` is not backed by a webhook source\", table),\n                        StatusCode::FORBIDDEN,\n                    )\n                })?\n                .clone();\n            (\n                webhook_source_info,\n                table_catalog.id(),\n                table_catalog.version_id().expect(\"table must be versioned\"),\n                row_id_index,\n                payload_schema,\n            )\n        };\n\n        let compute_client = choose_fast_insert_client(table_id, frontend_env, request_id)\n            .await\n            .map_err(|e| {\n                err(\n                    anyhow!(e).context(\"Failed to choose a compute node for fast insert\"),\n                    StatusCode::SERVICE_UNAVAILABLE,\n                )\n            })?;\n\n        Ok(WebhookTableInsertContext {\n            webhook_source_info,\n            table_id,\n            table_version_id,\n            row_id_index,\n            compute_client,\n            payload_schema,\n        })\n    }\n\n    async fn execute(\n        request: FastInsertRequest,\n        client: ComputeClient,\n    ) -> Result<FastInsertResponse> {","sourceCodeStart":263,"sourceCodeEnd":299,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/frontend/src/webhook/mod.rs#L263-L299","documentation":"The frontend could not select a healthy compute node to execute the webhook fast-insert path. `choose_fast_insert_client` consults the cluster catalog to find an available compute process for the table; if none can be chosen (no compute nodes registered, stale worker metadata, or client creation/connect failure), the request fails with HTTP 503 SERVICE_UNAVAILABLE. This indicates a cluster health/deployment problem, not a payload problem.","triggerScenarios":"POST to /webhook/{db}/{schema}/{table} when no compute node is reachable; all compute workers are failed/starting; the meta node's worker list is stale so the chosen node cannot be connected to; the ComputeClient gRPC handshake fails during construction.","commonSituations":"Sending webhook events during a rolling upgrade or compute restart; a compute node crashed and was not rescheduled; misconfigured compute listen/advertise addresses (e.g. risedev/k8s) so the frontend dials a wrong port; running with only meta+frontend while compute has not registered yet.","solutions":["Check cluster health and confirm at least one compute node is Running (risectl meta list-nodes or the meta dashboard).","Restart or reschedule the compute node(s): `./risedev d` locally or restart the compute Deployment in k8s.","Retry the webhook request once compute is healthy — it is a transient 503 and safe to retry with backoff.","Verify compute node advertise addresses/ports are reachable from the frontend node.","Inspect meta/compute logs for worker registration errors."],"exampleFix":"// before: fire-and-forget sender\npost(webhookUrl, body);\n// after: retry on 503\nlet res = await post(webhookUrl, body);\nif (res.status === 503) {\n  await sleep(backoff);\n  res = await post(webhookUrl, body);\n}","handlingStrategy":"retry","validationCode":"// ensure a healthy compute node exists before relying on webhooks\nconst nodes = await risectl.listComputeNodes();\nif (!nodes.some(n => n.state === 'RUNNING')) throw new Error('no compute node available');","typeGuard":null,"tryCatchPattern":"const res = await fetch(url, { method: 'POST', body });\nif (res.status === 503) {\n  await sleep(backoff(attempt));\n  return sendWithRetry(url, body, attempt + 1);\n}","preventionTips":["Monitor compute node liveness and alert on worker deregistration","Use idempotent webhook senders with exponential backoff","Avoid a zero-compute window during rolling upgrades","Verify advertise addresses/ports after network changes"],"tags":["cluster","compute-node","http-503","webhook"],"backgroundTag":"upstream-api-error","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}