{"record":{"id":"352c605c1ef5e5b3","repo":"apache/pulsar","slug":"failed-to-delete-service-for-function-s","errorCode":null,"errorMessage":"Failed to delete service for function %s","messagePattern":"Failed to delete service for function (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java","lineNumber":806,"sourceCode":"        AtomicBoolean success = new AtomicBoolean(false);\n        Actions.newBuilder()\n                .addAction(deleteService.toBuilder()\n                        .continueOn(true)\n                        .build())\n                .addAction(waitForServiceDeletion.toBuilder()\n                        .continueOn(false)\n                        .onSuccess(ignored -> success.set(true))\n                        .build())\n                .addAction(deleteService.toBuilder()\n                        .continueOn(true)\n                        .build())\n                .addAction(waitForServiceDeletion.toBuilder()\n                        .onSuccess(ignored -> success.set(true))\n                        .build())\n                .run();\n\n        if (!success.get()) {\n            throw new RuntimeException(String.format(\"Failed to delete service for function %s\", fqfn));\n        }\n    }\n\n    protected List<String> getExecutorCommand() {\n        List<String> cmds =\n                new ArrayList<>(getDownloadCommand(instanceConfig.getFunctionDetails(), originalCodeFileName, false));\n        if (isNotEmpty(originalTransformFunctionFileName)) {\n            cmds.add(\"&&\");\n            cmds.addAll(getDownloadCommand(instanceConfig.getFunctionDetails(),\n                originalTransformFunctionFileName, true));\n        }\n        cmds.add(\"&&\");\n        cmds.add(setShardIdEnvironmentVariableCommand());\n        cmds.add(\"&&\");\n        cmds.addAll(processArgs);\n        return Arrays.asList(\"sh\", \"-c\", String.join(\" \", cmds));\n    }\n","sourceCodeStart":788,"sourceCodeEnd":824,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java#L788-L824","documentation":"During KubernetesRuntime teardown (constructor-scoped deletion path / delete service action), the Action waiting for the function's Service to be deleted never succeeds, so a RuntimeException is thrown naming the function. The Service resource remains in the cluster.","triggerScenarios":"The delete-service action chain (delete call + waitForServiceDeletion) fails or times out — API errors, RBAC denial, or service stuck due to dependent resources (e.g. load balancer finalizers).","commonSituations":"Service of type LoadBalancer stuck in deletion with finalizers; API connectivity issues; missing RBAC delete permission on services.","solutions":["Check for finalizers on the Service: kubectl get svc <name> -n <ns> -o yaml and clear them if stuck.","Verify the worker ServiceAccount can delete services in the namespace.","Check API server connectivity and the root cause logged before this throw.","Delete the leftover service manually: kubectl delete svc <name> -n <ns>."],"exampleFix":"// before: stuck service\nkubectl patch svc <name> -n <ns> -p '{\"metadata\":{\"finalizers\":[]}}' --type=merge\n// after: kubectl delete svc <name> -n <ns>","handlingStrategy":"retry","validationCode":"// before stopping: check for leftover services with finalizers\nkubectl get svc -l <function-label-selector> -n <ns> -o jsonpath='{range .items[*]}{.metadata.name}{\":\"}{.metadata.finalizers}{\"\\n\"}{end}'","typeGuard":null,"tryCatchPattern":"try {\n    kubernetesRuntime.stop();\n} catch (RuntimeException e) {\n    if (e.getMessage().startsWith(\"Failed to delete service for function\")) {\n        // clear stuck finalizers or kubectl delete svc, then retry stop\n        log.warn(\"Service deletion failed; check finalizers\", e);\n    }\n    throw e;\n}","preventionTips":["Prefer ClusterIP services over LoadBalancer for functions to avoid LB finalizer delays","Ensure RBAC allows deleting services","Monitor for orphaned function services after stops","Keep API server connectivity healthy during worker shutdown"],"tags":["kubernetes","functions-runtime","service-deletion"],"backgroundTag":"kubernetes-resource-deletion-failed","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}