What causes "K8sJobExecutor fail to stop job" in apache/dolphinscheduler?

Verify the kubeconfig/k8sConfigMap connection works and the service account has get+delete on batch jobs in the target namespace (kubectl auth can-i delete jobs -n ).

apache/dolphinscheduler · error · TaskException

K8sJobExecutor fail to stop job

Error message

K8sJobExecutor fail to stop job

What it means

K8sTaskExecutor throws this TaskException when cancelling a running Kubernetes job fails. The stopJobOnK8s method checks whether the job exists via k8sUtils.jobExist and deletes it via k8sUtils.deleteJob; any exception from these Kubernetes API calls (existence check, delete call, connectivity) is wrapped as 'K8sJobExecutor fail to stop job'. It is thrown both from explicit cancelApplication and when a submit fails and cleanup is attempted.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/k8s/impl/K8sTaskExecutor.java:366

        } catch (Exception e) {
            log.error("[K8sJobExecutor-{}-{}] fail to submit job", taskName, taskInstanceId);
            throw new TaskException("K8sJobExecutor fail to submit job", e);
        }
    }

    @Override
    public void stopJobOnK8s(String k8sParameterStr) {
        K8sTaskMainParameters k8STaskMainParameters =
                JSONUtils.parseObject(k8sParameterStr, K8sTaskMainParameters.class);
        String namespaceName = k8STaskMainParameters.getNamespaceName();
        String jobName = job.getMetadata().getName();
        try {
            if (Boolean.TRUE.equals(k8sUtils.jobExist(jobName, namespaceName))) {
                k8sUtils.deleteJob(jobName, namespaceName);
            }
        } catch (Exception e) {
            log.error("[K8sJobExecutor-{}] fail to stop job", jobName);
            throw new TaskException("K8sJobExecutor fail to stop job", e);
        }
    }

    public int getK8sJobStatus(Job job) {
        JobStatus jobStatus = job.getStatus();
        if (jobStatus.getSucceeded() != null && jobStatus.getSucceeded() == 1) {
            return EXIT_CODE_SUCCESS;
        } else if (jobStatus.getFailed() != null && jobStatus.getFailed() == 1) {
            return EXIT_CODE_FAILURE;
        } else {
            return TaskConstants.RUNNING_CODE;
        }
    }

    public void setTaskStatus(int jobStatus, String taskInstanceId, TaskResponse taskResponse) {
        if (jobStatus == EXIT_CODE_SUCCESS || jobStatus == EXIT_CODE_FAILURE) {
            if (jobStatus == EXIT_CODE_SUCCESS) {
                log.info("[K8sJobExecutor-{}] succeed in k8s", job.getMetadata().getName());

View on GitHub (pinned to 02eac45a1b)

Solutions

Verify the kubeconfig/k8sConfigMap connection works and the service account has get+delete on batch jobs in the target namespace (kubectl auth can-i delete jobs -n <ns>).
Check the job/namespace actually exist: kubectl get job <jobName> -n <namespace>; adjust namespaceName/jobName if they mismatch.
Retry the cancellation — the Kubernetes client may have hit a transient network error to the API server.
If the job is stuck in Terminating, clear finalizers or wait for termination before deleting, or delete with propagation policy via kubectl first.
Check worker logs for the inner exception (cause of TaskException) to distinguish RBAC 403 vs connection vs not-found causes.

Example fix

// before
k8sUtils.deleteJob(jobName, namespaceName);
// after
try {
    if (Boolean.TRUE.equals(k8sUtils.jobExist(jobName, namespaceName))) {
        k8sUtils.deleteJobsCollection(namespaceName, jobName, null); // explicit propagation policy
    }
} catch (KubernetesClientException e) {
    log.warn("delete job {} ns {} returned {}, may already be gone", jobName, namespaceName, e.getCode());
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (!k8sUtils.jobExist(jobName, namespaceName)) { log.warn("job already gone"); return; }
// pre-check RBAC once at startup: kubectl auth can-i delete jobs.batch -n <ns>

Try / catch

try { stopJobOnK8s(jobName, namespace); } catch (TaskException e) { log.error("cancel failed for {}", jobName, e); /* mark task as kill-failed, do not lose the cause */ }

Prevention

Validate kubeconfig/namespace at task startup, not at cancel time
Grant the service account delete on jobs.batch before deploying
Retry once on transient KubernetesClientException from the API server
Monitor for jobs stuck in Terminating and clear finalizers

When it happens

Trigger: Calling cancelApplication on a K8s job whose namespace is wrong or job already terminated mid-check; k8sConfigMap/connection invalid so the Kubernetes API client throws; deleteJob hitting an API error (forbidden RBAC, job in terminating/child-owner conflict); transient network failure to the API server during jobExist or deleteJob.

Common situations: Worker cancellation during a network blip to the API server; service account lacking delete permission on batch/jobs; operator deleted the namespace or job manually while the task was being cancelled; stale/expired kubeconfig after cluster migration.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/1c3265513aebaae8. Report an issue: GitHub.