apache/pulsar · error · RuntimeException

Failed to delete secrets for function %s

Error message

Failed to delete secrets for function %s

What it means

KubernetesSecretsTokenAuthProvider.cleanUpAuthData runs a kubernetes-client action list that deletes the function's auth secrets and waits via an AtomicBoolean success flag. If the deletion actions do not report success before the wait completes, it throws RuntimeException 'Failed to delete secrets for function <fqfn>'. Indicates Kubernetes API delete operations failed or timed out.

Source

Thrown at pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/KubernetesSecretsTokenAuthProvider.java:247

        AtomicBoolean success = new AtomicBoolean(false);
        Actions.newBuilder()
                .addAction(deleteSecrets.toBuilder()
                        .continueOn(true)
                        .build())
                .addAction(waitForSecretsDeletion.toBuilder()
                        .continueOn(false)
                        .onSuccess(ignore -> success.set(true))
                        .build())
                .addAction(deleteSecrets.toBuilder()
                        .continueOn(true)
                        .build())
                .addAction(waitForSecretsDeletion.toBuilder()
                        .onSuccess(ignore -> success.set(true))
                        .build())
                .run();

        if (!success.get()) {
            throw new RuntimeException(String.format("Failed to delete secrets for function %s", fqfn));
        }
    }

    @Override
    public Optional<FunctionAuthData> updateAuthData(FunctionDetails funcDetails,
                                                     Optional<FunctionAuthData> existingFunctionAuthData,
                                                     AuthenticationDataSource authenticationDataSource)
            throws Exception {

        String secretId;
        secretId = existingFunctionAuthData.map(functionAuthData -> new String(functionAuthData.getData()))
                .orElseGet(() -> {
                    @SuppressWarnings("deprecation")
                    String id = RandomStringUtils.random(5, true, true).toLowerCase();
                    return id;
                });

        String token;

View on GitHub (pinned to 820761864e)

Solutions

  1. Check RBAC: grant the function worker service account 'delete' on secrets in the namespace
  2. Verify connectivity to the Kubernetes API server and valid credentials/kubeconfig
  3. Confirm the namespace and secret names exist / inspect events with kubectl describe
  4. Retry the deletion; if secrets are already absent, treat cleanup as idempotent

Example fix

// before
kubectl get rolebindings -n <ns>  # worker can't delete secrets
// after
kubectl create role secrets-cleaner --verb=delete,create,get --resource=secrets -n <ns>
kubectl create rolebinding worker-secrets-cleaner --role=secrets-cleaner \
  --serviceaccount=<ns>:function-worker -n <ns>
Defensive patterns

Strategy: retry

Validate before calling

// verify RBAC before running functions
// kubectl auth can-i delete secrets --as=system:serviceaccount:<ns>:function-worker -n <ns>

Try / catch

try {
    provider.cleanUpAuthData(fqfn, namespace);
} catch (RuntimeException e) {
    if (e.getMessage().startsWith("Failed to delete secrets")) {
        // check kube API connectivity/RBAC; retry cleanup (idempotent)
    }
}

Prevention

When it happens

Trigger: Kubernetes API errors during secret deletion (RBAC denied, connection failure, namespace missing), or the action runner's success flag never set within the wait window.

Common situations: Function worker service account lacking delete permission on secrets; API server unreachable from the worker; secrets already gone causing unexpected action results; kubeconfig misconfiguration.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/ffd15eeaed52a62b. Report an issue: GitHub.