{"record":{"id":"1c3265513aebaae8","repo":"apache/dolphinscheduler","slug":"k8sjobexecutor-fail-to-stop-job","errorCode":null,"errorMessage":"K8sJobExecutor fail to stop job","messagePattern":"K8sJobExecutor fail to stop job","errorType":"exception","errorClass":"TaskException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/k8s/impl/K8sTaskExecutor.java","lineNumber":366,"sourceCode":"        } catch (Exception e) {\n            log.error(\"[K8sJobExecutor-{}-{}] fail to submit job\", taskName, taskInstanceId);\n            throw new TaskException(\"K8sJobExecutor fail to submit job\", e);\n        }\n    }\n\n    @Override\n    public void stopJobOnK8s(String k8sParameterStr) {\n        K8sTaskMainParameters k8STaskMainParameters =\n                JSONUtils.parseObject(k8sParameterStr, K8sTaskMainParameters.class);\n        String namespaceName = k8STaskMainParameters.getNamespaceName();\n        String jobName = job.getMetadata().getName();\n        try {\n            if (Boolean.TRUE.equals(k8sUtils.jobExist(jobName, namespaceName))) {\n                k8sUtils.deleteJob(jobName, namespaceName);\n            }\n        } catch (Exception e) {\n            log.error(\"[K8sJobExecutor-{}] fail to stop job\", jobName);\n            throw new TaskException(\"K8sJobExecutor fail to stop job\", e);\n        }\n    }\n\n    public int getK8sJobStatus(Job job) {\n        JobStatus jobStatus = job.getStatus();\n        if (jobStatus.getSucceeded() != null && jobStatus.getSucceeded() == 1) {\n            return EXIT_CODE_SUCCESS;\n        } else if (jobStatus.getFailed() != null && jobStatus.getFailed() == 1) {\n            return EXIT_CODE_FAILURE;\n        } else {\n            return TaskConstants.RUNNING_CODE;\n        }\n    }\n\n    public void setTaskStatus(int jobStatus, String taskInstanceId, TaskResponse taskResponse) {\n        if (jobStatus == EXIT_CODE_SUCCESS || jobStatus == EXIT_CODE_FAILURE) {\n            if (jobStatus == EXIT_CODE_SUCCESS) {\n                log.info(\"[K8sJobExecutor-{}] succeed in k8s\", job.getMetadata().getName());","sourceCodeStart":348,"sourceCodeEnd":384,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/k8s/impl/K8sTaskExecutor.java#L348-L384","documentation":"K8sTaskExecutor throws this TaskException when cancelling a running Kubernetes job fails. The stopJobOnK8s method checks whether the job exists via k8sUtils.jobExist and deletes it via k8sUtils.deleteJob; any exception from these Kubernetes API calls (existence check, delete call, connectivity) is wrapped as 'K8sJobExecutor fail to stop job'. It is thrown both from explicit cancelApplication and when a submit fails and cleanup is attempted.","triggerScenarios":"Calling cancelApplication on a K8s job whose namespace is wrong or job already terminated mid-check; k8sConfigMap/connection invalid so the Kubernetes API client throws; deleteJob hitting an API error (forbidden RBAC, job in terminating/child-owner conflict); transient network failure to the API server during jobExist or deleteJob.","commonSituations":"Worker cancellation during a network blip to the API server; service account lacking delete permission on batch/jobs; operator deleted the namespace or job manually while the task was being cancelled; stale/expired kubeconfig after cluster migration.","solutions":["Verify the kubeconfig/k8sConfigMap connection works and the service account has get+delete on batch jobs in the target namespace (kubectl auth can-i delete jobs -n <ns>).","Check the job/namespace actually exist: kubectl get job <jobName> -n <namespace>; adjust namespaceName/jobName if they mismatch.","Retry the cancellation — the Kubernetes client may have hit a transient network error to the API server.","If the job is stuck in Terminating, clear finalizers or wait for termination before deleting, or delete with propagation policy via kubectl first.","Check worker logs for the inner exception (cause of TaskException) to distinguish RBAC 403 vs connection vs not-found causes."],"exampleFix":"// before\nk8sUtils.deleteJob(jobName, namespaceName);\n// after\ntry {\n    if (Boolean.TRUE.equals(k8sUtils.jobExist(jobName, namespaceName))) {\n        k8sUtils.deleteJobsCollection(namespaceName, jobName, null); // explicit propagation policy\n    }\n} catch (KubernetesClientException e) {\n    log.warn(\"delete job {} ns {} returned {}, may already be gone\", jobName, namespaceName, e.getCode());\n}","handlingStrategy":"try-catch","validationCode":"if (!k8sUtils.jobExist(jobName, namespaceName)) { log.warn(\"job already gone\"); return; }\n// pre-check RBAC once at startup: kubectl auth can-i delete jobs.batch -n <ns>","typeGuard":null,"tryCatchPattern":"try { stopJobOnK8s(jobName, namespace); } catch (TaskException e) { log.error(\"cancel failed for {}\", jobName, e); /* mark task as kill-failed, do not lose the cause */ }","preventionTips":["Validate kubeconfig/namespace at task startup, not at cancel time","Grant the service account delete on jobs.batch before deploying","Retry once on transient KubernetesClientException from the API server","Monitor for jobs stuck in Terminating and clear finalizers"],"tags":["kubernetes","cancellation","api-client"],"backgroundTag":"http-error-response","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}