apache/pulsar · critical · RuntimeException

Failed to delete service for function %s

Error message

Failed to delete service for function %s

What it means

During KubernetesRuntime teardown (constructor-scoped deletion path / delete service action), the Action waiting for the function's Service to be deleted never succeeds, so a RuntimeException is thrown naming the function. The Service resource remains in the cluster.

Source

Thrown at pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java:806

        AtomicBoolean success = new AtomicBoolean(false);
        Actions.newBuilder()
                .addAction(deleteService.toBuilder()
                        .continueOn(true)
                        .build())
                .addAction(waitForServiceDeletion.toBuilder()
                        .continueOn(false)
                        .onSuccess(ignored -> success.set(true))
                        .build())
                .addAction(deleteService.toBuilder()
                        .continueOn(true)
                        .build())
                .addAction(waitForServiceDeletion.toBuilder()
                        .onSuccess(ignored -> success.set(true))
                        .build())
                .run();

        if (!success.get()) {
            throw new RuntimeException(String.format("Failed to delete service for function %s", fqfn));
        }
    }

    protected List<String> getExecutorCommand() {
        List<String> cmds =
                new ArrayList<>(getDownloadCommand(instanceConfig.getFunctionDetails(), originalCodeFileName, false));
        if (isNotEmpty(originalTransformFunctionFileName)) {
            cmds.add("&&");
            cmds.addAll(getDownloadCommand(instanceConfig.getFunctionDetails(),
                originalTransformFunctionFileName, true));
        }
        cmds.add("&&");
        cmds.add(setShardIdEnvironmentVariableCommand());
        cmds.add("&&");
        cmds.addAll(processArgs);
        return Arrays.asList("sh", "-c", String.join(" ", cmds));
    }

View on GitHub (pinned to 820761864e)

Solutions

1. Check for finalizers on the Service: kubectl get svc <name> -n <ns> -o yaml and clear them if stuck.
2. Verify the worker ServiceAccount can delete services in the namespace.
3. Check API server connectivity and the root cause logged before this throw.
4. Delete the leftover service manually: kubectl delete svc <name> -n <ns>.

Example fix

// before: stuck service
kubectl patch svc <name> -n <ns> -p '{"metadata":{"finalizers":[]}}' --type=merge
// after: kubectl delete svc <name> -n <ns>

// before stopping: check for leftover services with finalizers
kubectl get svc -l <function-label-selector> -n <ns> -o jsonpath='{range .items[*]}{.metadata.name}{":"}{.metadata.finalizers}{"\n"}{end}'

try {
    kubernetesRuntime.stop();
} catch (RuntimeException e) {
    if (e.getMessage().startsWith("Failed to delete service for function")) {
        // clear stuck finalizers or kubectl delete svc, then retry stop
        log.warn("Service deletion failed; check finalizers", e);
    }
    throw e;
}

When it happens

Trigger: The delete-service action chain (delete call + waitForServiceDeletion) fails or times out — API errors, RBAC denial, or service stuck due to dependent resources (e.g. load balancer finalizers).

Common situations: Service of type LoadBalancer stuck in deletion with finalizers; API connectivity issues; missing RBAC delete permission on services.

Related errors

• Invalid value for BROKER_CLIENT_TRUST_CERTS_SECRET_NAME. Exp
• Invalid value for SERVICE_ACCOUNT_TOKEN_EXPIRATION_SECONDS.
• Invalid value for SERVICE_ACCOUNT_TOKEN_AUDIENCE. Expected a
• Failed to create service for function %s
• Failed to create statefulset for function %s

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/352c605c1ef5e5b3. Report an issue: GitHub.