apache/pulsar · critical · RuntimeException

Failed to create service for function %s

Error message

Failed to create service for function %s

What it means

KubernetesRuntime.submitService() runs a retry Action to create the function's Kubernetes Service via the k8s client; if the Action never succeeds (all retries exhausted), it throws RuntimeException naming the fully-qualified function name. This means the function worker could not register the Service backing the function instance.

Source

Thrown at pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java:505

                                .success(false)
                                .errorMsg(errorMsg)
                                .build();
                    }

                    return Actions.ActionResult.builder().success(true).build();
                })
                .build();


        AtomicBoolean success = new AtomicBoolean(false);
        Actions.newBuilder()
                .addAction(createService.toBuilder()
                        .onSuccess(ignored -> success.set(true))
                        .build())
                .run();

        if (!success.get()) {
            throw new RuntimeException(String.format("Failed to create service for function %s", fqfn));
        }
    }

    @VisibleForTesting
    V1Service createService() {
        final String jobName = createJobName(instanceConfig.getFunctionDetails(), this.jobName);

        final V1Service service = new V1Service();

        // setup stateful set metadata
        final V1ObjectMeta objectMeta = new V1ObjectMeta();
        objectMeta.name(jobName);
        objectMeta.setLabels(getLabels(instanceConfig.getFunctionDetails()));
        // we don't technically need to set this, but it is useful for testing
        objectMeta.setNamespace(jobNamespace);
        service.metadata(objectMeta);

        // create the stateful set spec

View on GitHub (pinned to 820761864e)

Solutions

1. Check the underlying logged exception for the k8s API failure cause (connection refused, 403 Forbidden, 422).
2. Verify the functions worker's ServiceAccount has RBAC (create services in the namespace).
3. Confirm the Kubernetes API endpoint and credentials (kubeconfig / in-cluster config) are correct.
4. Ensure the service name derived from the function name doesn't conflict with an existing resource.

Example fix

// before (RBAC missing) -> verify rolebinding
kubectl auth can-i create services -n <ns> --as=system:serviceaccount:<ns>:<worker-sa>
// after: grant it
kubectl create rolebinding fn-worker-svc --role=pulsar-fn-runtime --serviceaccount=<ns>:<worker-sa> -n <ns>

// before starting the runtime
if (!kubectlAuthCanICreate("services", namespace)) {
    throw new IllegalStateException("Worker ServiceAccount lacks permission to create services in " + namespace);
}

try {
    kubernetesRuntime.start();
} catch (RuntimeException e) {
    if (e.getMessage().startsWith("Failed to create service for function")) {
        // inspect worker logs for the k8s client root cause, check API server health and RBAC
        log.error("K8s service creation failed for function; root cause in earlier logs", e);
    }
    throw e;
}

When it happens

Trigger: submitService() called from start() when the Kubernetes API create-service call fails repeatedly — e.g. API server unreachable, auth/RBAC denied, invalid service spec, or name conflicts.

Common situations: Kubernetes API server down or misconfigured k8s client credentials; the function worker's ServiceAccount lacks RBAC permission to create Services; cluster DNS/namespace issues.

Related errors

• Invalid value for BROKER_CLIENT_TRUST_CERTS_SECRET_NAME. Exp
• Invalid value for SERVICE_ACCOUNT_TOKEN_EXPIRATION_SECONDS.
• Invalid value for SERVICE_ACCOUNT_TOKEN_AUDIENCE. Expected a
• Failed to create statefulset for function %s
• Failed to delete statefulset for function %s

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/977473795841f348. Report an issue: GitHub.