{"record":{"id":"da7c097d66816175","repo":"apache/dolphinscheduler","slug":"fail-to-register-batch-job-watcher","errorCode":null,"errorMessage":"fail to register batch job watcher","messagePattern":"fail to register batch job watcher","errorType":"exception","errorClass":"TaskException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/K8sUtils.java","lineNumber":82,"sourceCode":"\n    public Boolean jobExist(String jobName, String namespace) {\n        try {\n            Job job = client.batch().v1().jobs().inNamespace(namespace).withName(jobName).get();\n            return job != null;\n        } catch (Exception e) {\n            throw new TaskException(\"fail to check job: \", e);\n        }\n    }\n\n    public Watch createBatchJobWatcher(String jobName, Watcher<Job> watcher) {\n        try {\n            return client.batch()\n                    .v1()\n                    .jobs()\n                    .withName(jobName)\n                    .watch(watcher);\n        } catch (Exception e) {\n            throw new TaskException(\"fail to register batch job watcher\", e);\n        }\n    }\n\n    public String getPodLog(String jobName, String namespace) {\n        try {\n            List<Pod> podList = client.pods().inNamespace(namespace).list().getItems();\n            String podName = null;\n            for (Pod pod : podList) {\n                podName = pod.getMetadata().getName();\n                if (podName.contains(\"-\") && jobName.equals(podName.substring(0, podName.lastIndexOf(\"-\")))) {\n                    break;\n                }\n            }\n            return client.pods().inNamespace(namespace)\n                    .withName(podName)\n                    .tailingLines(LOG_LINES)\n                    .getLog(Boolean.TRUE);\n        } catch (Exception e) {","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/K8sUtils.java#L64-L100","documentation":"K8sUtils.createBatchJobWatcher() registers a Watch on a named Batch/V1 Job and wraps any failure in a TaskException 'fail to register batch job watcher'. Watch registration is an HTTP long-poll against the API server, so network issues and RBAC are the typical root causes.","triggerScenarios":"Calling createBatchJobWatcher(jobName, watcher) when the watch connection cannot be established: API server unreachable, kubeconfig invalid, missing watch permission on batch/jobs, or the resource watch endpoint rejected the request.","commonSituations":"K8s task log collection failing on clusters with restricted RBAC (no watch verb); proxy/firewall cutting long-lived connections; TLS certificate issues; client built with expired credentials.","solutions":["Check the chained cause for KubernetesClientException details (403 vs connection error).","Verify RBAC includes 'watch' on batch/jobs in the target namespace.","Ensure network path allows long-lived HTTPS connections to the API server (no aggressive proxy timeouts).","Rebuild the client with fresh kubeconfig if credentials expired."],"exampleFix":"// before\nWatch watch = k8sUtils.createBatchJobWatcher(jobName, watcher);\n// after\nWatch watch = null;\ntry {\n    watch = k8sUtils.createBatchJobWatcher(jobName, watcher);\n} catch (TaskException e) {\n    log.warn(\"watch registration failed, falling back to polling\", e);\n    pollJobStatus(jobName, namespace);\n}","handlingStrategy":"fallback","validationCode":"if (!apiServerReachable()) {\n    throw new IllegalStateException(\"cannot register watch: API server unreachable\");\n}","typeGuard":"boolean hasWatchPermission(KubernetesClient client, String ns) {\n    try { client.batch().v1().jobs().inNamespace(ns).withName(\"__probe__\").watch(l -> {}).close(); return true; }\n    catch (Exception e) { return false; }\n}","tryCatchPattern":"try {\n    watch = k8sUtils.createBatchJobWatcher(jobName, watcher);\n} catch (TaskException e) {\n    log.warn(\"watch failed, falling back to polling\", e);\n    startPolling(jobName, namespace, watcher);\n}","preventionTips":["Grant watch permission on batch/jobs to the worker service account.","Allow long-lived HTTPS connections through proxies/firewalls to the API server.","Implement a polling fallback for watch registration failures."],"tags":["kubernetes","watch","task-exception"],"backgroundTag":"http-error-response","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}