apache/pulsar · critical · RuntimeException

Failed to delete statefulset for function %s

Error message

Failed to delete statefulset for function %s

What it means

deleteStatefulSet() runs a retry Action that deletes the StatefulSet and waits for deletion; if the action chain never signals success, it throws RuntimeException naming the function. Deletion failure leaves stale function pods/statefulset in the cluster.

Source

Thrown at pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java:719

        AtomicBoolean success = new AtomicBoolean(false);
        Actions.newBuilder()
                .addAction(deleteStatefulSet.toBuilder()
                        .continueOn(true)
                        .build())
                .addAction(waitForStatefulSetDeletion.toBuilder()
                        .continueOn(false)
                        .onSuccess(ignored -> success.set(true))
                        .build())
                .addAction(deleteStatefulSet.toBuilder()
                        .continueOn(true)
                        .build())
                .addAction(waitForStatefulSetDeletion.toBuilder()
                        .onSuccess(ignored -> success.set(true))
                        .build())
                .run();

        if (!success.get()) {
            throw new RuntimeException(String.format("Failed to delete statefulset for function %s", fqfn));
        } else {
            // wait for pods to terminate
            Actions.newBuilder()
                    .addAction(waitForStatefulPodsToTerminate)
                    .run();
        }
    }

    public void deleteService() throws InterruptedException {
        String fqfn = FunctionCommon.getFullyQualifiedName(instanceConfig.getFunctionDetails());
        String serviceName = createJobName(instanceConfig.getFunctionDetails(), this.jobName);

        Actions.Action deleteService = Actions.Action.builder()
                .actionName(String.format("Deleting service for function %s", fqfn))
                .numRetries(KubernetesRuntimeFactory.numRetries)
                .sleepBetweenInvocationsMs(KubernetesRuntimeFactory.sleepBetweenRetriesMs)
                .supplier(() -> {
                    try {

View on GitHub (pinned to 820761864e)

Solutions

1. Check whether the StatefulSet has stuck finalizers: kubectl get sts -n <ns> and inspect metadata.finalizers; remove blocking finalizers.
2. Verify RBAC delete permissions for statefulsets for the worker's ServiceAccount.
3. Confirm the API server is reachable and check for the underlying exception logged before this throw.
4. Manually delete the stale StatefulSet and restart the function.

Example fix

// before: stuck finalizer
kubectl patch statefulset <name> -n <ns> -p '{"metadata":{"finalizers":[]}}' --type=merge
// after: retry function stop/delete

// before stopping: detect stuck finalizers
kubectl get statefulset <job-name> -n <ns> -o jsonpath='{.metadata.finalizers}'

try {
    kubernetesRuntime.stop();
} catch (RuntimeException e) {
    if (e.getMessage().startsWith("Failed to delete statefulset for function")) {
        // clear finalizers or delete manually, then retry
        log.warn("StatefulSet deletion failed; check finalizers and RBAC", e);
    }
    throw e;
}

When it happens

Trigger: Called by stop() (or tests) when the k8s delete call keeps failing or the StatefulSet remains visible past the retry deadline — API errors, finalizers blocking deletion, RBAC denial.

Common situations: Stuck finalizers on the StatefulSet; API server unreachable; worker lacks delete permissions; namespace terminating.

Related errors

• Failed to create statefulset for function %s
• Invalid value for BROKER_CLIENT_TRUST_CERTS_SECRET_NAME. Exp
• Invalid value for SERVICE_ACCOUNT_TOKEN_EXPIRATION_SECONDS.
• Invalid value for SERVICE_ACCOUNT_TOKEN_AUDIENCE. Expected a
• Failed to create service for function %s

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/20464c44faeb16a7. Report an issue: GitHub.