apache/dolphinscheduler · error · TaskException

fail to delete job

Error message

fail to delete job

What it means

K8sUtils.deleteJob() deletes a Batch/V1 Job by name in a namespace and wraps any client exception in a TaskException 'fail to delete job'. Any failure in the delete call — connectivity, RBAC, or API server errors — surfaces under this generic message.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/K8sUtils.java:61

                    .v1()
                    .jobs()
                    .inNamespace(namespace)
                    .create(job);
        } catch (Exception e) {
            throw new TaskException("fail to create job", e);
        }
    }

    public void deleteJob(String jobName, String namespace) {
        try {
            client.batch()
                    .v1()
                    .jobs()
                    .inNamespace(namespace)
                    .withName(jobName)
                    .delete();
        } catch (Exception e) {
            throw new TaskException("fail to delete job", e);
        }
    }

    public Boolean jobExist(String jobName, String namespace) {
        try {
            Job job = client.batch().v1().jobs().inNamespace(namespace).withName(jobName).get();
            return job != null;
        } catch (Exception e) {
            throw new TaskException("fail to check job: ", e);
        }
    }

    public Watch createBatchJobWatcher(String jobName, Watcher<Job> watcher) {
        try {
            return client.batch()
                    .v1()
                    .jobs()
                    .withName(jobName)

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Read the chained cause in worker logs for the actual Fabric8 KubernetesClientException.
2. Confirm cluster reachability and a valid kubeconfig on the worker node.
3. Check RBAC allows 'delete' on batch/jobs in the target namespace.
4. If the job is stuck, delete with propagation or clear finalizers via kubectl, then retry.
5. Verify the namespace is active (not Terminating) before deleting jobs.

Example fix

// before
k8sUtils.deleteJob(jobName, namespace); // throws on stuck job
// after
try {
    k8sUtils.deleteJob(jobName, namespace);
} catch (TaskException e) {
    log.warn("job cleanup failed, checking existence", e);
    if (Boolean.TRUE.equals(k8sUtils.jobExist(jobName, namespace))) {
        throw e;
    }
}

if (!Boolean.TRUE.equals(k8sUtils.jobExist(jobName, namespace))) {
    return; // nothing to delete
}

boolean canDeleteJobs(KubernetesClient client, String ns) {
    return client != null && client.namespaces().withName(ns).get() != null;
}

try {
    k8sUtils.deleteJob(jobName, namespace);
} catch (TaskException e) {
    if (Boolean.TRUE.equals(k8sUtils.jobExist(jobName, namespace))) {
        throw e; // still there, real failure
    }
    log.warn("delete errored but job is gone; treating as success");
}

When it happens

Trigger: Calling deleteJob(jobName, namespace) when the API server rejects or cannot process the DELETE: unreachable cluster, no 'jobs delete' RBAC permission, invalid kubeconfig, or a finalizer conflict preventing deletion.

Common situations: Cleanup of a completed/failed K8s task job when the cluster was replaced or kubeconfig rotated; service account permissions narrowed; job stuck with finalizers; namespace already being terminated.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• fail to create job
• fail to check job:
• fail to register batch job watcher
• namespace %s does not exist in k8s cluster, please create na
• fail to get k8s ApiClient:%s

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/301502151903ce92. Report an issue: GitHub.