{"record":{"id":"979b2b4a80777f7b","repo":"apache/dolphinscheduler","slug":"failed-to-get-kubernetes-application-status","errorCode":null,"errorMessage":"Failed to get Kubernetes application status","messagePattern":"Failed to get Kubernetes application status","errorType":"exception","errorClass":"TaskException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/KubernetesApplicationManager.java","lineNumber":179,"sourceCode":"     */\n    private TaskExecutionStatus getApplicationStatus(KubernetesApplicationManagerContext kubernetesApplicationManagerContext,\n                                                     FilterWatchListDeletable<Pod, PodList, PodResource> watchList) throws TaskException {\n        String phase;\n        try {\n            if (Objects.isNull(watchList)) {\n                watchList = getListenPod(kubernetesApplicationManagerContext);\n            }\n            List<Pod> driverPod = watchList.list().getItems();\n            if (!driverPod.isEmpty()) {\n                // cluster mode\n                Pod driver = driverPod.get(0);\n                phase = driver.getStatus().getPhase();\n            } else {\n                // client mode\n                phase = FINISH;\n            }\n        } catch (Exception e) {\n            throw new TaskException(\"Failed to get Kubernetes application status\", e);\n        }\n\n        return phase.equals(FAILED) || phase.equals(UNKNOWN) ? TaskExecutionStatus.FAILURE\n                : TaskExecutionStatus.SUCCESS;\n    }\n\n    /**\n     * get pod's log watcher\n     *\n     * @param kubernetesApplicationManagerContext\n     * @return\n     */\n    @SneakyThrows\n    public LogWatch getPodLogWatcher(KubernetesApplicationManagerContext kubernetesApplicationManagerContext) {\n        KubernetesClient client = getClient(kubernetesApplicationManagerContext);\n        boolean podIsReady = false;\n        Pod pod = null;\n        while (!podIsReady) {","sourceCodeStart":161,"sourceCodeEnd":197,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/am/KubernetesApplicationManager.java#L161-L197","documentation":"This TaskException is thrown by KubernetesApplicationManager.getApplicationStatus when the call to the Kubernetes API to fetch a Spark/other driver pod's status fails with any Exception (client construction, API request, or pod lookup). The application manager wraps the low-level failure (e.g. KubernetesClientException) into a TaskException with this message, preserving the cause. It is raised both when polling application status and when killing an application that needs its status resolved.","triggerScenarios":"Calling killApplication() or getApplicationStatus() while the task is configured with applicationManager=KUBERNETES; the KubernetesClient build fails (bad kube config / master URL), the driver pod no longer exists (deleted/evicted, causing a 404), the API server is unreachable, RBAC denies pod get, or client mode is misconfigured so driver.getStatus() errors.","commonSituations":"kubeconfig or KUBECONFIG/service-account token missing or invalid in the worker environment; driver pod already cleaned up by Spark before status poll; network/firewall blocking the Kubernetes API server; RBAC role lacking get pods permission; wrong kubernetes master URL in Spark conf; short-lived pods vanishing between job submit and status query.","solutions":["Verify the driver pod exists: kubectl get pods -n <namespace> and confirm the pod name/label selector used by the manager matches; recreate the task if the pod was already deleted.","Check Kubernetes connectivity and credentials on the worker: validate the kubeconfig / in-cluster service account with kubectl (or a KubernetesClient) from the same host/container.","Fix RBAC: grant the worker's ServiceAccount get access to pods (and logs) in the task namespace.","Confirm the Spark conf kubernetes master URL and namespace in the task definition are correct and reachable from the worker.","Inspect the wrapped cause (KubernetesClientException) in the TaskException stack trace for the HTTP code to pinpoint 401/403/404/timeout and fix accordingly."],"exampleFix":"// before: failure aborts the task\nTaskExecutionStatus status = applicationManager.getApplicationStatus(applicationId);\n\n// after: tolerate already-terminated pods\ntry {\n    status = applicationManager.getApplicationStatus(applicationId);\n} catch (TaskException e) {\n    log.warn(\"Driver pod status unavailable, assuming finished: {}\", e.getMessage());\n    status = TaskExecutionStatus.KILL;\n}","handlingStrategy":"try-catch","validationCode":"// Pre-check Kubernetes connectivity before running the task\ntry (KubernetesClient client = new KubernetesClientBuilder().build()) {\n    client.pods().inNamespace(namespace).withLabels(driverLabels).list();\n    // reachable + authorized; also confirm pod exists:\n    boolean exists = client.pods().inNamespace(namespace)\n            .withName(podName).get() != null;\n    if (!exists) throw new IllegalStateException(\"Driver pod not found: \" + podName);\n} catch (KubernetesClientException e) {\n    throw new IllegalStateException(\"Kubernetes API not reachable/authorized: \" + e.getCode(), e);\n}","typeGuard":"static boolean isDriverPodPresent(KubernetesClient client, String ns, String podName) {\n    return podName != null && client.pods().inNamespace(ns).withName(podName).get() != null;\n}","tryCatchPattern":"try {\n    status = applicationManager.getApplicationStatus(applicationId);\n} catch (TaskException e) {\n    Throwable cause = e.getCause();\n    log.error(\"K8s status fetch failed, cause class={}, msg={}\",\n            cause == null ? \"null\" : cause.getClass().getSimpleName(),\n            cause == null ? \"\" : cause.getMessage());\n    status = TaskExecutionStatus.FAILURE; // or KILL for already-terminated pods\n}","preventionTips":["Validate kubeconfig / in-cluster service account on the worker host before launching Kubernetes-managed tasks.","Grant the worker ServiceAccount get pods permissions (RBAC) in the task namespace.","Keep driver pods alive long enough for status polling; avoid aggressive TTL/cleanup settings that delete the pod before status is read.","Confirm the kubernetes master URL and namespace in Spark conf are reachable from the worker network."],"tags":["kubernetes","task-execution","status-polling","cluster-connectivity"],"backgroundTag":"http-request-failed","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}