apache/pulsar · error · RuntimeException

Failed to upsert authentication secret for function %s/%s/%s

Error message

Failed to upsert authentication secret for function %s/%s/%s

What it means

upsertSecret builds kubernetes actions to create/update the function's authentication secret and waits on a success flag; if the upsert actions don't succeed it throws RuntimeException 'Failed to upsert authentication secret for function <tenant>/<namespace>/<name>'. This blocks updateAuthData from persisting function auth material.

Source

Thrown at pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/KubernetesSecretsTokenAuthProvider.java:348

                        return Actions.ActionResult.builder()
                                .success(false)
                                .errorMsg(errorMsg)
                                .build();
                    }

                    return Actions.ActionResult.builder().success(true).build();
                })
                .build();

        AtomicBoolean success = new AtomicBoolean(false);
        Actions.newBuilder()
                .addAction(createAuthSecret.toBuilder()
                        .onSuccess(ignore -> success.set(true))
                        .build())
                .run();

        if (!success.get()) {
            throw new RuntimeException(
                    String.format("Failed to upsert authentication secret for function %s/%s/%s", tenant, namespace,
                            name));
        }
    }

    private String createSecret(String token, FunctionDetails funcDetails)
            throws ApiException, InterruptedException {
        String kubeNamespace = getKubeNamespace(funcDetails);
        String tenant = funcDetails.getTenant();
        String namespace = funcDetails.getNamespace();
        String name = funcDetails.getName();

        StringBuilder sb = new StringBuilder();
        Actions.Action createAuthSecret = Actions.Action.builder()
                .actionName(
                        String.format("Creating authentication secret for function %s/%s/%s", tenant, namespace, name))
                .numRetries(NUM_RETRIES)
                .sleepBetweenInvocationsMs(SLEEP_BETWEEN_RETRIES_MS)

View on GitHub (pinned to 820761864e)

Solutions

Grant the function worker RBAC create/update/get on secrets in the target namespace
Check the Kubernetes API server reachability and auth from the worker pod
Inspect secret size; reduce function auth data if it exceeds the ~1MB limit
Check worker logs/events (kubectl describe pod, kubectl get events) for the underlying API error and retry

Example fix

// before
# worker service account has only 'get' on secrets
// after
kubectl create role secret-writer --verb=create,update,get --resource=secrets -n <ns>
kubectl create rolebinding fn-worker-secret-writer --role=secret-writer \
  --serviceaccount=<ns>:function-worker -n <ns>

Defensive patterns

Strategy: retry

Validate before calling

// preflight
// kubectl auth can-i create secrets --as=system:serviceaccount:<ns>:function-worker -n <ns>
// and verify function auth data size < ~1MB

Try / catch

try {
    provider.updateAuthData(funcDetails, existing, secretsProvider);
} catch (RuntimeException e) {
    if (e.getMessage().startsWith("Failed to upsert authentication secret")) {
        // inspect underlying k8s error in logs; fix RBAC/size; retry
    }
}

Prevention

Grant create/update on secrets to the function worker SA
Keep function auth data small (etcd ~1MB secret limit)
Monitor k8s API health from worker pods

When it happens

Trigger: Kubernetes API failure during secret create/update (RBAC lacking create/update on secrets, API unreachable, invalid secret payload e.g. token too large) within updateAuthData.

Common situations: Service account missing update/create RBAC on secrets; oversized auth data exceeding etcd's 1MB secret limit; API server connectivity issues from function worker pods.

Understand the failure class

Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/4ddfc4623f171588. Report an issue: GitHub.