{"record":{"id":"ffd15eeaed52a62b","repo":"apache/pulsar","slug":"failed-to-delete-secrets-for-function-s","errorCode":null,"errorMessage":"Failed to delete secrets for function %s","messagePattern":"Failed to delete secrets for function (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/KubernetesSecretsTokenAuthProvider.java","lineNumber":247,"sourceCode":"        AtomicBoolean success = new AtomicBoolean(false);\n        Actions.newBuilder()\n                .addAction(deleteSecrets.toBuilder()\n                        .continueOn(true)\n                        .build())\n                .addAction(waitForSecretsDeletion.toBuilder()\n                        .continueOn(false)\n                        .onSuccess(ignore -> success.set(true))\n                        .build())\n                .addAction(deleteSecrets.toBuilder()\n                        .continueOn(true)\n                        .build())\n                .addAction(waitForSecretsDeletion.toBuilder()\n                        .onSuccess(ignore -> success.set(true))\n                        .build())\n                .run();\n\n        if (!success.get()) {\n            throw new RuntimeException(String.format(\"Failed to delete secrets for function %s\", fqfn));\n        }\n    }\n\n    @Override\n    public Optional<FunctionAuthData> updateAuthData(FunctionDetails funcDetails,\n                                                     Optional<FunctionAuthData> existingFunctionAuthData,\n                                                     AuthenticationDataSource authenticationDataSource)\n            throws Exception {\n\n        String secretId;\n        secretId = existingFunctionAuthData.map(functionAuthData -> new String(functionAuthData.getData()))\n                .orElseGet(() -> {\n                    @SuppressWarnings(\"deprecation\")\n                    String id = RandomStringUtils.random(5, true, true).toLowerCase();\n                    return id;\n                });\n\n        String token;","sourceCodeStart":229,"sourceCodeEnd":265,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/KubernetesSecretsTokenAuthProvider.java#L229-L265","documentation":"KubernetesSecretsTokenAuthProvider.cleanUpAuthData runs a kubernetes-client action list that deletes the function's auth secrets and waits via an AtomicBoolean success flag. If the deletion actions do not report success before the wait completes, it throws RuntimeException 'Failed to delete secrets for function <fqfn>'. Indicates Kubernetes API delete operations failed or timed out.","triggerScenarios":"Kubernetes API errors during secret deletion (RBAC denied, connection failure, namespace missing), or the action runner's success flag never set within the wait window.","commonSituations":"Function worker service account lacking delete permission on secrets; API server unreachable from the worker; secrets already gone causing unexpected action results; kubeconfig misconfiguration.","solutions":["Check RBAC: grant the function worker service account 'delete' on secrets in the namespace","Verify connectivity to the Kubernetes API server and valid credentials/kubeconfig","Confirm the namespace and secret names exist / inspect events with kubectl describe","Retry the deletion; if secrets are already absent, treat cleanup as idempotent"],"exampleFix":"// before\nkubectl get rolebindings -n <ns>  # worker can't delete secrets\n// after\nkubectl create role secrets-cleaner --verb=delete,create,get --resource=secrets -n <ns>\nkubectl create rolebinding worker-secrets-cleaner --role=secrets-cleaner \\\n  --serviceaccount=<ns>:function-worker -n <ns>","handlingStrategy":"retry","validationCode":"// verify RBAC before running functions\n// kubectl auth can-i delete secrets --as=system:serviceaccount:<ns>:function-worker -n <ns>","typeGuard":null,"tryCatchPattern":"try {\n    provider.cleanUpAuthData(fqfn, namespace);\n} catch (RuntimeException e) {\n    if (e.getMessage().startsWith(\"Failed to delete secrets\")) {\n        // check kube API connectivity/RBAC; retry cleanup (idempotent)\n    }\n}","preventionTips":["Pre-provision RBAC for secret delete on the worker service account","Ensure kubeconfig/credentials are valid in worker pods","Treat secret cleanup as idempotent and retry on failure"],"tags":["kubernetes","secrets","cleanup"],"backgroundTag":"kubernetes-secret-operation-failed","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}