apache/dolphinscheduler · error · TaskException

Failed to kill Kubernetes application with label

Error message

Failed to kill Kubernetes application with label 

What it means

KubernetesApplicationManager.killApplication wraps exceptions from deleting the pod watch/resources into TaskException('Failed to kill Kubernetes application with label ' + labelValue). The kill of the K8s application (e.g. Spark driver pod) via label selector failed.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/KubernetesApplicationManager.java:82

    private final Map<String, KubernetesClient> cacheClientMap = new ConcurrentHashMap<>();

    @Override
    public boolean killApplication(KubernetesApplicationManagerContext kubernetesApplicationManagerContext) throws TaskException {

        boolean isKill;
        String labelValue = kubernetesApplicationManagerContext.getLabelValue();
        FilterWatchListDeletable<Pod, PodList, PodResource> watchList =
                getListenPod(kubernetesApplicationManagerContext);
        try {
            if (getApplicationStatus(kubernetesApplicationManagerContext, watchList).isFailure()) {
                log.error("Driver pod is in FAILED or UNKNOWN status.");
                isKill = false;
            } else {
                watchList.delete();
                isKill = true;
            }
        } catch (Exception e) {
            throw new TaskException("Failed to kill Kubernetes application with label " + labelValue, e);
        } finally {
            // remove client cache after killing application
            removeCache(labelValue);
        }

        return isKill;
    }

    @Override
    public ResourceManagerType getResourceManagerType() {
        return ResourceManagerType.KUBERNETES;
    }

    /**
     * get driver pod
     *
     * @param kubernetesApplicationManagerContext
     * @return

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Check the wrapped cause for the real client exception (401/403 vs connection refused)
2. Verify RBAC: the worker's service account needs delete on pods and the task resources
3. Confirm kubeconfig/connection settings in the K8s task execution context
4. Delete the pod manually with kubectl delete pod -l <label> to unblock the task
5. Check connectivity from worker to the K8s API server

// pre-flight: verify API access and RBAC
KubernetesClient client = buildClient(k8sExecutionContext);
client.pods().inNamespace(namespace).withLabel("app", labelValue).list(); // throws early on 401/403

try {
    boolean killed = applicationManager.killApplication(appIds);
} catch (TaskException e) {
    log.error("K8s kill failed for label {}: {}", labelValue, e.getCause().getMessage(), e);
    // fallback: kubectl delete pod -l <label> or fix RBAC
}

When it happens

Trigger: The kubernetesClient call(s) inside the try block throw — API server unreachable, RBAC denies delete, watch/list by label fails, or the delete call errors while the task is cancelled.

Common situations: Worker kubeconfig/credentials invalid or expired; service account lacks delete/pods permission; K8s API server temporarily unavailable; wrong namespace/label so the delete targets nothing and the watch call fails; network policy blocking worker->apiserver.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• namespace %s does not exist in k8s cluster, please create na
• fail to get k8s ApiClient:%s
• K8S_NAMESPACE_EXIST
• CLUSTER_NOT_EXISTS
• K8S_CLIENT_OPS_ERROR

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/6d2a634561e3f403. Report an issue: GitHub.