{"record":{"id":"977473795841f348","repo":"apache/pulsar","slug":"failed-to-create-service-for-function-s","errorCode":null,"errorMessage":"Failed to create service for function %s","messagePattern":"Failed to create service for function (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java","lineNumber":505,"sourceCode":"                                .success(false)\n                                .errorMsg(errorMsg)\n                                .build();\n                    }\n\n                    return Actions.ActionResult.builder().success(true).build();\n                })\n                .build();\n\n\n        AtomicBoolean success = new AtomicBoolean(false);\n        Actions.newBuilder()\n                .addAction(createService.toBuilder()\n                        .onSuccess(ignored -> success.set(true))\n                        .build())\n                .run();\n\n        if (!success.get()) {\n            throw new RuntimeException(String.format(\"Failed to create service for function %s\", fqfn));\n        }\n    }\n\n    @VisibleForTesting\n    V1Service createService() {\n        final String jobName = createJobName(instanceConfig.getFunctionDetails(), this.jobName);\n\n        final V1Service service = new V1Service();\n\n        // setup stateful set metadata\n        final V1ObjectMeta objectMeta = new V1ObjectMeta();\n        objectMeta.name(jobName);\n        objectMeta.setLabels(getLabels(instanceConfig.getFunctionDetails()));\n        // we don't technically need to set this, but it is useful for testing\n        objectMeta.setNamespace(jobNamespace);\n        service.metadata(objectMeta);\n\n        // create the stateful set spec","sourceCodeStart":487,"sourceCodeEnd":523,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java#L487-L523","documentation":"KubernetesRuntime.submitService() runs a retry Action to create the function's Kubernetes Service via the k8s client; if the Action never succeeds (all retries exhausted), it throws RuntimeException naming the fully-qualified function name. This means the function worker could not register the Service backing the function instance.","triggerScenarios":"submitService() called from start() when the Kubernetes API create-service call fails repeatedly — e.g. API server unreachable, auth/RBAC denied, invalid service spec, or name conflicts.","commonSituations":"Kubernetes API server down or misconfigured k8s client credentials; the function worker's ServiceAccount lacks RBAC permission to create Services; cluster DNS/namespace issues.","solutions":["Check the underlying logged exception for the k8s API failure cause (connection refused, 403 Forbidden, 422).","Verify the functions worker's ServiceAccount has RBAC (create services in the namespace).","Confirm the Kubernetes API endpoint and credentials (kubeconfig / in-cluster config) are correct.","Ensure the service name derived from the function name doesn't conflict with an existing resource."],"exampleFix":"// before (RBAC missing) -> verify rolebinding\nkubectl auth can-i create services -n <ns> --as=system:serviceaccount:<ns>:<worker-sa>\n// after: grant it\nkubectl create rolebinding fn-worker-svc --role=pulsar-fn-runtime --serviceaccount=<ns>:<worker-sa> -n <ns>","handlingStrategy":"retry","validationCode":"// before starting the runtime\nif (!kubectlAuthCanICreate(\"services\", namespace)) {\n    throw new IllegalStateException(\"Worker ServiceAccount lacks permission to create services in \" + namespace);\n}","typeGuard":null,"tryCatchPattern":"try {\n    kubernetesRuntime.start();\n} catch (RuntimeException e) {\n    if (e.getMessage().startsWith(\"Failed to create service for function\")) {\n        // inspect worker logs for the k8s client root cause, check API server health and RBAC\n        log.error(\"K8s service creation failed for function; root cause in earlier logs\", e);\n    }\n    throw e;\n}","preventionTips":["Pre-provision RBAC roles allowing service/statefulset creation for the worker ServiceAccount","Monitor Kubernetes API server availability from the functions worker","Keep function names RFC-1123 compliant to avoid spec validation failures","Check namespace quotas before deploying functions"],"tags":["kubernetes","functions-runtime","service-creation"],"backgroundTag":"kubernetes-resource-creation-failed","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}