apache/dolphinscheduler · critical · TaskException

K8sJobExecutor fail to submit job

Error message

K8sJobExecutor fail to submit job

What it means

K8sTaskExecutor.submitJob2k8s() first deletes any leftover job (stopJobOnK8s) and then calls k8sUtils.createJob(namespaceName, job) to create the Job on the cluster. Any exception from the Kubernetes API client or job construction is logged and rethrown as TaskException("K8sJobExecutor fail to submit job", e). The task instance will fail with exitStatusCode -1 upstream.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/k8s/impl/K8sTaskExecutor.java:350

        }
    }

    @Override
    public void submitJob2k8s(String k8sParameterStr) {
        int taskInstanceId = taskRequest.getTaskInstanceId();
        String taskName = taskRequest.getTaskName().toLowerCase(Locale.ROOT);
        K8sTaskMainParameters k8STaskMainParameters =
                JSONUtils.parseObject(k8sParameterStr, K8sTaskMainParameters.class);
        try {
            log.info("[K8sJobExecutor-{}-{}] start to submit job", taskName, taskInstanceId);
            buildK8sJob(k8STaskMainParameters);
            stopJobOnK8s(k8sParameterStr);
            String namespaceName = k8STaskMainParameters.getNamespaceName();
            k8sUtils.createJob(namespaceName, job);
            log.info("[K8sJobExecutor-{}-{}] submitted job successfully", taskName, taskInstanceId);
        } catch (Exception e) {
            log.error("[K8sJobExecutor-{}-{}] fail to submit job", taskName, taskInstanceId);
            throw new TaskException("K8sJobExecutor fail to submit job", e);
        }
    }

    @Override
    public void stopJobOnK8s(String k8sParameterStr) {
        K8sTaskMainParameters k8STaskMainParameters =
                JSONUtils.parseObject(k8sParameterStr, K8sTaskMainParameters.class);
        String namespaceName = k8STaskMainParameters.getNamespaceName();
        String jobName = job.getMetadata().getName();
        try {
            if (Boolean.TRUE.equals(k8sUtils.jobExist(jobName, namespaceName))) {
                k8sUtils.deleteJob(jobName, namespaceName);
            }
        } catch (Exception e) {
            log.error("[K8sJobExecutor-{}] fail to stop job", jobName);
            throw new TaskException("K8sJobExecutor fail to stop job", e);
        }
    }

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Read the caused-by KubernetesClientException for the HTTP status/reason (403 RBAC, 404 namespace, 422 invalid spec).
2. Verify the namespace exists and the worker's service account has rights to create jobs in it.
3. Check the generated Job spec: job name legality (lowercase RFC1123), image name, nodeSelector/affinity entries from K8sTaskMainParameters.
4. Confirm cluster connectivity/credentials on the worker (kubeconfig, API server address).
5. Ensure the image exists and pull secrets are configured for private registries.

Example fix

// before
String namespaceName = k8STaskMainParameters.getNamespaceName(); // may not exist
k8sUtils.createJob(namespaceName, job);
// after
String namespaceName = k8STaskMainParameters.getNamespaceName();
if (!k8sUtils.namespaceExists(namespaceName)) {
    throw new TaskException("Namespace not found: " + namespaceName);
}
k8sUtils.createJob(namespaceName, job);

// before submitting
if (!k8sUtils.namespaceExists(k8STaskMainParameters.getNamespaceName())) {
    throw new IllegalArgumentException("Namespace does not exist: " + k8STaskMainParameters.getNamespaceName());
}
if (!job.getMetadata().getName().matches("[a-z0-9]([-a-z0-9]*[a-z0-9])?")) {
    throw new IllegalArgumentException("Illegal (non-RFC1123) job name: " + job.getMetadata().getName());
}

try {
    k8sUtils.createJob(namespaceName, job);
} catch (KubernetesClientException e) {
    log.error("K8s API rejected job creation: code={}, message={}", e.getCode(), e.getMessage());
    if (e.getCode() == 403) {
        throw new TaskException("RBAC denied: grant jobs/create on namespace " + namespaceName, e);
    }
    throw new TaskException("K8sJobExecutor fail to submit job", e);
}

When it happens

Trigger: The Kubernetes API rejects createJob: namespace does not exist, RBAC forbids jobs/create, invalid Job spec (illegal job/container name, bad image, malformed affinity/nodeSelector built from k8STaskMainParameters), or the API server is unreachable.

Common situations: Wrong namespaceName in task params; worker service account lacking RBAC permissions; invalid job name (uppercase/illegal chars from taskName); nonexistent or private image without pull secrets; cluster connection misconfig (master URL/credentials).

Related errors

• Backfill workflow failed: %s
• Recover workflow instance failed: %s
• namespace %s does not exist in k8s cluster, please create na
• fail to get k8s ApiClient:%s
• K8S_NAMESPACE_EXIST

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/fd99e2dff4fde718. Report an issue: GitHub.