{"record":{"id":"20464c44faeb16a7","repo":"apache/pulsar","slug":"failed-to-delete-statefulset-for-function-s","errorCode":null,"errorMessage":"Failed to delete statefulset for function %s","messagePattern":"Failed to delete statefulset for function (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java","lineNumber":719,"sourceCode":"        AtomicBoolean success = new AtomicBoolean(false);\n        Actions.newBuilder()\n                .addAction(deleteStatefulSet.toBuilder()\n                        .continueOn(true)\n                        .build())\n                .addAction(waitForStatefulSetDeletion.toBuilder()\n                        .continueOn(false)\n                        .onSuccess(ignored -> success.set(true))\n                        .build())\n                .addAction(deleteStatefulSet.toBuilder()\n                        .continueOn(true)\n                        .build())\n                .addAction(waitForStatefulSetDeletion.toBuilder()\n                        .onSuccess(ignored -> success.set(true))\n                        .build())\n                .run();\n\n        if (!success.get()) {\n            throw new RuntimeException(String.format(\"Failed to delete statefulset for function %s\", fqfn));\n        } else {\n            // wait for pods to terminate\n            Actions.newBuilder()\n                    .addAction(waitForStatefulPodsToTerminate)\n                    .run();\n        }\n    }\n\n    public void deleteService() throws InterruptedException {\n        String fqfn = FunctionCommon.getFullyQualifiedName(instanceConfig.getFunctionDetails());\n        String serviceName = createJobName(instanceConfig.getFunctionDetails(), this.jobName);\n\n        Actions.Action deleteService = Actions.Action.builder()\n                .actionName(String.format(\"Deleting service for function %s\", fqfn))\n                .numRetries(KubernetesRuntimeFactory.numRetries)\n                .sleepBetweenInvocationsMs(KubernetesRuntimeFactory.sleepBetweenRetriesMs)\n                .supplier(() -> {\n                    try {","sourceCodeStart":701,"sourceCodeEnd":737,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java#L701-L737","documentation":"deleteStatefulSet() runs a retry Action that deletes the StatefulSet and waits for deletion; if the action chain never signals success, it throws RuntimeException naming the function. Deletion failure leaves stale function pods/statefulset in the cluster.","triggerScenarios":"Called by stop() (or tests) when the k8s delete call keeps failing or the StatefulSet remains visible past the retry deadline — API errors, finalizers blocking deletion, RBAC denial.","commonSituations":"Stuck finalizers on the StatefulSet; API server unreachable; worker lacks delete permissions; namespace terminating.","solutions":["Check whether the StatefulSet has stuck finalizers: kubectl get sts -n <ns> and inspect metadata.finalizers; remove blocking finalizers.","Verify RBAC delete permissions for statefulsets for the worker's ServiceAccount.","Confirm the API server is reachable and check for the underlying exception logged before this throw.","Manually delete the stale StatefulSet and restart the function."],"exampleFix":"// before: stuck finalizer\nkubectl patch statefulset <name> -n <ns> -p '{\"metadata\":{\"finalizers\":[]}}' --type=merge\n// after: retry function stop/delete","handlingStrategy":"retry","validationCode":"// before stopping: detect stuck finalizers\nkubectl get statefulset <job-name> -n <ns> -o jsonpath='{.metadata.finalizers}'","typeGuard":null,"tryCatchPattern":"try {\n    kubernetesRuntime.stop();\n} catch (RuntimeException e) {\n    if (e.getMessage().startsWith(\"Failed to delete statefulset for function\")) {\n        // clear finalizers or delete manually, then retry\n        log.warn(\"StatefulSet deletion failed; check finalizers and RBAC\", e);\n    }\n    throw e;\n}","preventionTips":["Avoid custom controllers adding finalizers to function statefulsets","Ensure worker RBAC includes delete on statefulsets and pods","Watch for terminating namespaces which block deletion","Alert on functions stuck in stopping state"],"tags":["kubernetes","functions-runtime","statefulset","deletion"],"backgroundTag":"kubernetes-resource-deletion-failed","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}