{"record":{"id":"8d997a3b19e29418","repo":"risingwavelabs/risingwave","slug":"serverless-backfill-controller-returned-error","errorCode":null,"errorMessage":"serverless backfill controller returned error","messagePattern":"serverless backfill controller returned error","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/stream/stream_manager.rs","lineNumber":559,"sourceCode":"            );\n        }\n\n        let request = tonic::Request::new(ProvisionRequest {});\n        let mut client =\n            node_group_controller_service_client::NodeGroupControllerServiceClient::connect(\n                sbc_addr.clone(),\n            )\n            .await\n            .with_context(|| {\n                format!(\n                    \"unable to reach serverless backfill controller at addr {}\",\n                    sbc_addr\n                )\n            })?;\n\n        match client.provision(request).await {\n            Ok(resp) => Ok(resp.into_inner().resource_group),\n            Err(e) => Err(anyhow::Error::new(TonicStatusWrapper::new(e))\n                .context(\"serverless backfill controller returned error\")\n                .into()),\n        }\n    }\n\n    async fn finalize_create_streaming_job_resource_group(\n        &self,\n        resource_type: &streaming_job_resource_type::ResourceType,\n        streaming_job_model: &mut streaming_job::Model,\n    ) -> MetaResult<()> {\n        if !matches!(\n            resource_type,\n            streaming_job_resource_type::ResourceType::ServerlessBackfill(true)\n        ) {\n            return Ok(());\n        }\n\n        let group = self.provision_serverless_backfill_resource_group().await?;","sourceCodeStart":541,"sourceCodeEnd":577,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/stream/stream_manager.rs#L541-L577","documentation":"The meta node failed to provision the serverless backfill resource group because the gRPC call to the Serverless Backfill Controller (SBC) returned a Tonic status error. The meta node wraps the tonic Status into TonicStatusWrapper and adds this context so operators know the failure happened while talking to the external SBC service, not inside RisingWave itself.","triggerScenarios":"Calling client.provision(request) on the SBC gRPC client returns Err(Status) — e.g. SBC endpoint unreachable (sbc_addr misconfigured), SBC not deployed, SBC crashed, request deadline exceeded, or SBC rejected the provisioning request (invalid resource group config).","commonSituations":"Deploying RisingWave with serverless backfill enabled but the SBC address (sbc_addr) pointing to a wrong host/port; SBC service not yet healthy during cluster startup; network/firewall blocking meta→SBC traffic; SBC version mismatch rejecting the ProvisionRequest.","solutions":["Verify the serverless backfill controller address (sbc_addr) in meta configuration is correct and reachable (curl/grpc probe the endpoint).","Check the SBC service logs/health; restart or redeploy it if it is down or crashed.","Inspect the wrapped TonicStatusWrapper source in the anyhow report for the concrete gRPC status (Unavailable, DeadlineExceeded, InvalidArgument) and fix accordingly.","If SBC is intentionally absent, disable serverless backfill so provision is not called."],"exampleFix":"// before\nmatch client.provision(request).await {\n    Ok(resp) => Ok(resp.into_inner().resource_group),\n    Err(e) => Err(anyhow::Error::new(TonicStatusWrapper::new(e))\n        .context(\"serverless backfill controller returned error\")\n        .into()),\n}\n// after\nmatch client.provision(request).await {\n    Ok(resp) => Ok(resp.into_inner().resource_group),\n    Err(e) => {\n        tracing::warn!(status = ?e, \"provision request to SBC failed\");\n        Err(anyhow::Error::new(TonicStatusWrapper::new(e))\n            .context(format!(\"serverless backfill controller at {sbc_addr} returned error\"))\n            .into())\n    }\n}","handlingStrategy":"retry","validationCode":"// before enabling serverless backfill\ntonic::transport::Endpoint::from_shared(sbc_addr.clone())?.connect().await?;","typeGuard":"fn is_sbc_provision_err(e: &anyhow::Error) -> bool {\n    e.chain().any(|c| c.downcast_ref::<String>().map(|s| s.contains(\"serverless backfill controller\")).unwrap_or(false))\n}","tryCatchPattern":"match res {\n    Err(e) if is_retryable_tonic(&e) => retry_with_backoff(3, || provision(...)).await,\n    Err(e) => return Err(e),\n    Ok(r) => Ok(r),\n}","preventionTips":["Health-check the SBC endpoint before cluster start and gate serverless backfill on it.","Pin and validate sbc_addr in the meta config; fail fast at startup if unreachable.","Add retries with backoff on Unavailable/DeadlineExceeded statuses.","Alert on SBC service health in deployment monitoring."],"tags":["grpc","network","meta-node","backfill"],"backgroundTag":"upstream-api-error","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}