apache/dolphinscheduler · error · TaskException

Failed to get Kubernetes application status

Error message

Failed to get Kubernetes application status

What it means

This TaskException is thrown by KubernetesApplicationManager.getApplicationStatus when the call to the Kubernetes API to fetch a Spark/other driver pod's status fails with any Exception (client construction, API request, or pod lookup). The application manager wraps the low-level failure (e.g. KubernetesClientException) into a TaskException with this message, preserving the cause. It is raised both when polling application status and when killing an application that needs its status resolved.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/KubernetesApplicationManager.java:179

     */
    private TaskExecutionStatus getApplicationStatus(KubernetesApplicationManagerContext kubernetesApplicationManagerContext,
                                                     FilterWatchListDeletable<Pod, PodList, PodResource> watchList) throws TaskException {
        String phase;
        try {
            if (Objects.isNull(watchList)) {
                watchList = getListenPod(kubernetesApplicationManagerContext);
            }
            List<Pod> driverPod = watchList.list().getItems();
            if (!driverPod.isEmpty()) {
                // cluster mode
                Pod driver = driverPod.get(0);
                phase = driver.getStatus().getPhase();
            } else {
                // client mode
                phase = FINISH;
            }
        } catch (Exception e) {
            throw new TaskException("Failed to get Kubernetes application status", e);
        }

        return phase.equals(FAILED) || phase.equals(UNKNOWN) ? TaskExecutionStatus.FAILURE
                : TaskExecutionStatus.SUCCESS;
    }

    /**
     * get pod's log watcher
     *
     * @param kubernetesApplicationManagerContext
     * @return
     */
    @SneakyThrows
    public LogWatch getPodLogWatcher(KubernetesApplicationManagerContext kubernetesApplicationManagerContext) {
        KubernetesClient client = getClient(kubernetesApplicationManagerContext);
        boolean podIsReady = false;
        Pod pod = null;
        while (!podIsReady) {

View on GitHub (pinned to 02eac45a1b)

Solutions

  1. Verify the driver pod exists: kubectl get pods -n <namespace> and confirm the pod name/label selector used by the manager matches; recreate the task if the pod was already deleted.
  2. Check Kubernetes connectivity and credentials on the worker: validate the kubeconfig / in-cluster service account with kubectl (or a KubernetesClient) from the same host/container.
  3. Fix RBAC: grant the worker's ServiceAccount get access to pods (and logs) in the task namespace.
  4. Confirm the Spark conf kubernetes master URL and namespace in the task definition are correct and reachable from the worker.
  5. Inspect the wrapped cause (KubernetesClientException) in the TaskException stack trace for the HTTP code to pinpoint 401/403/404/timeout and fix accordingly.

Example fix

// before: failure aborts the task
TaskExecutionStatus status = applicationManager.getApplicationStatus(applicationId);

// after: tolerate already-terminated pods
try {
    status = applicationManager.getApplicationStatus(applicationId);
} catch (TaskException e) {
    log.warn("Driver pod status unavailable, assuming finished: {}", e.getMessage());
    status = TaskExecutionStatus.KILL;
}
Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-check Kubernetes connectivity before running the task
try (KubernetesClient client = new KubernetesClientBuilder().build()) {
    client.pods().inNamespace(namespace).withLabels(driverLabels).list();
    // reachable + authorized; also confirm pod exists:
    boolean exists = client.pods().inNamespace(namespace)
            .withName(podName).get() != null;
    if (!exists) throw new IllegalStateException("Driver pod not found: " + podName);
} catch (KubernetesClientException e) {
    throw new IllegalStateException("Kubernetes API not reachable/authorized: " + e.getCode(), e);
}

Type guard

static boolean isDriverPodPresent(KubernetesClient client, String ns, String podName) {
    return podName != null && client.pods().inNamespace(ns).withName(podName).get() != null;
}

Try / catch

try {
    status = applicationManager.getApplicationStatus(applicationId);
} catch (TaskException e) {
    Throwable cause = e.getCause();
    log.error("K8s status fetch failed, cause class={}, msg={}",
            cause == null ? "null" : cause.getClass().getSimpleName(),
            cause == null ? "" : cause.getMessage());
    status = TaskExecutionStatus.FAILURE; // or KILL for already-terminated pods
}

Prevention

When it happens

Trigger: Calling killApplication() or getApplicationStatus() while the task is configured with applicationManager=KUBERNETES; the KubernetesClient build fails (bad kube config / master URL), the driver pod no longer exists (deleted/evicted, causing a 404), the API server is unreachable, RBAC denies pod get, or client mode is misconfigured so driver.getStatus() errors.

Common situations: kubeconfig or KUBECONFIG/service-account token missing or invalid in the worker environment; driver pod already cleaned up by Spark before status poll; network/firewall blocking the Kubernetes API server; RBAC role lacking get pods permission; wrong kubernetes master URL in Spark conf; short-lived pods vanishing between job submit and status query.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/979b2b4a80777f7b. Report an issue: GitHub.