{"record":{"id":"6d2a634561e3f403","repo":"apache/dolphinscheduler","slug":"failed-to-kill-kubernetes-application-with-label","errorCode":null,"errorMessage":"Failed to kill Kubernetes application with label ","messagePattern":"Failed to kill Kubernetes application with label ","errorType":"exception","errorClass":"TaskException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/KubernetesApplicationManager.java","lineNumber":82,"sourceCode":"    private final Map<String, KubernetesClient> cacheClientMap = new ConcurrentHashMap<>();\n\n    @Override\n    public boolean killApplication(KubernetesApplicationManagerContext kubernetesApplicationManagerContext) throws TaskException {\n\n        boolean isKill;\n        String labelValue = kubernetesApplicationManagerContext.getLabelValue();\n        FilterWatchListDeletable<Pod, PodList, PodResource> watchList =\n                getListenPod(kubernetesApplicationManagerContext);\n        try {\n            if (getApplicationStatus(kubernetesApplicationManagerContext, watchList).isFailure()) {\n                log.error(\"Driver pod is in FAILED or UNKNOWN status.\");\n                isKill = false;\n            } else {\n                watchList.delete();\n                isKill = true;\n            }\n        } catch (Exception e) {\n            throw new TaskException(\"Failed to kill Kubernetes application with label \" + labelValue, e);\n        } finally {\n            // remove client cache after killing application\n            removeCache(labelValue);\n        }\n\n        return isKill;\n    }\n\n    @Override\n    public ResourceManagerType getResourceManagerType() {\n        return ResourceManagerType.KUBERNETES;\n    }\n\n    /**\n     * get driver pod\n     *\n     * @param kubernetesApplicationManagerContext\n     * @return","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/KubernetesApplicationManager.java#L64-L100","documentation":"KubernetesApplicationManager.killApplication wraps exceptions from deleting the pod watch/resources into TaskException('Failed to kill Kubernetes application with label ' + labelValue). The kill of the K8s application (e.g. Spark driver pod) via label selector failed.","triggerScenarios":"The kubernetesClient call(s) inside the try block throw — API server unreachable, RBAC denies delete, watch/list by label fails, or the delete call errors while the task is cancelled.","commonSituations":"Worker kubeconfig/credentials invalid or expired; service account lacks delete/pods permission; K8s API server temporarily unavailable; wrong namespace/label so the delete targets nothing and the watch call fails; network policy blocking worker->apiserver.","solutions":["Check the wrapped cause for the real client exception (401/403 vs connection refused)","Verify RBAC: the worker's service account needs delete on pods and the task resources","Confirm kubeconfig/connection settings in the K8s task execution context","Delete the pod manually with kubectl delete pod -l <label> to unblock the task","Check connectivity from worker to the K8s API server"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// pre-flight: verify API access and RBAC\nKubernetesClient client = buildClient(k8sExecutionContext);\nclient.pods().inNamespace(namespace).withLabel(\"app\", labelValue).list(); // throws early on 401/403","typeGuard":null,"tryCatchPattern":"try {\n    boolean killed = applicationManager.killApplication(appIds);\n} catch (TaskException e) {\n    log.error(\"K8s kill failed for label {}: {}\", labelValue, e.getCause().getMessage(), e);\n    // fallback: kubectl delete pod -l <label> or fix RBAC\n}","preventionTips":["Grant the worker service account delete permission on pods","Verify kubeconfig/connection before task submission","Watch for expired credentials/tokens on long-running workers","Have a manual kubectl-by-label cleanup runbook for failed kills"],"tags":["kubernetes","kill","pod-cleanup"],"backgroundTag":"http-error-response","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}