risingwavelabs/risingwave · error

serverless backfill controller returned error

Error message

serverless backfill controller returned error

What it means

The meta node failed to provision the serverless backfill resource group because the gRPC call to the Serverless Backfill Controller (SBC) returned a Tonic status error. The meta node wraps the tonic Status into TonicStatusWrapper and adds this context so operators know the failure happened while talking to the external SBC service, not inside RisingWave itself.

Source

Thrown at src/meta/src/stream/stream_manager.rs:559

            );
        }

        let request = tonic::Request::new(ProvisionRequest {});
        let mut client =
            node_group_controller_service_client::NodeGroupControllerServiceClient::connect(
                sbc_addr.clone(),
            )
            .await
            .with_context(|| {
                format!(
                    "unable to reach serverless backfill controller at addr {}",
                    sbc_addr
                )
            })?;

        match client.provision(request).await {
            Ok(resp) => Ok(resp.into_inner().resource_group),
            Err(e) => Err(anyhow::Error::new(TonicStatusWrapper::new(e))
                .context("serverless backfill controller returned error")
                .into()),
        }
    }

    async fn finalize_create_streaming_job_resource_group(
        &self,
        resource_type: &streaming_job_resource_type::ResourceType,
        streaming_job_model: &mut streaming_job::Model,
    ) -> MetaResult<()> {
        if !matches!(
            resource_type,
            streaming_job_resource_type::ResourceType::ServerlessBackfill(true)
        ) {
            return Ok(());
        }

        let group = self.provision_serverless_backfill_resource_group().await?;

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify the serverless backfill controller address (sbc_addr) in meta configuration is correct and reachable (curl/grpc probe the endpoint).
2. Check the SBC service logs/health; restart or redeploy it if it is down or crashed.
3. Inspect the wrapped TonicStatusWrapper source in the anyhow report for the concrete gRPC status (Unavailable, DeadlineExceeded, InvalidArgument) and fix accordingly.
4. If SBC is intentionally absent, disable serverless backfill so provision is not called.

Example fix

// before
match client.provision(request).await {
    Ok(resp) => Ok(resp.into_inner().resource_group),
    Err(e) => Err(anyhow::Error::new(TonicStatusWrapper::new(e))
        .context("serverless backfill controller returned error")
        .into()),
}
// after
match client.provision(request).await {
    Ok(resp) => Ok(resp.into_inner().resource_group),
    Err(e) => {
        tracing::warn!(status = ?e, "provision request to SBC failed");
        Err(anyhow::Error::new(TonicStatusWrapper::new(e))
            .context(format!("serverless backfill controller at {sbc_addr} returned error"))
            .into())
    }
}

// before enabling serverless backfill
tonic::transport::Endpoint::from_shared(sbc_addr.clone())?.connect().await?;

fn is_sbc_provision_err(e: &anyhow::Error) -> bool {
    e.chain().any(|c| c.downcast_ref::<String>().map(|s| s.contains("serverless backfill controller")).unwrap_or(false))
}

match res {
    Err(e) if is_retryable_tonic(&e) => retry_with_backoff(3, || provision(...)).await,
    Err(e) => return Err(e),
    Ok(r) => Ok(r),
}

When it happens

Trigger: Calling client.provision(request) on the SBC gRPC client returns Err(Status) — e.g. SBC endpoint unreachable (sbc_addr misconfigured), SBC not deployed, SBC crashed, request deadline exceeded, or SBC rejected the provisioning request (invalid resource group config).

Common situations: Deploying RisingWave with serverless backfill enabled but the SBC address (sbc_addr) pointing to a wrong host/port; SBC service not yet healthy during cluster startup; network/firewall blocking meta→SBC traffic; SBC version mismatch rejecting the ProvisionRequest.

Related errors

• bigquery insert error: end of resp stream
• Cannot open client to compute node {addr:?}
• RpcError
• failed to send request to {} {:?}
• Cannot alter the job {} because it is blocked by creating un

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/8d997a3b19e29418. Report an issue: GitHub.