apache/dolphinscheduler · error · TaskException

fail to register batch job watcher

Error message

fail to register batch job watcher

What it means

K8sUtils.createBatchJobWatcher() registers a Watch on a named Batch/V1 Job and wraps any failure in a TaskException 'fail to register batch job watcher'. Watch registration is an HTTP long-poll against the API server, so network issues and RBAC are the typical root causes.

Solutions

1. Check the chained cause for KubernetesClientException details (403 vs connection error).
2. Verify RBAC includes 'watch' on batch/jobs in the target namespace.
3. Ensure network path allows long-lived HTTPS connections to the API server (no aggressive proxy timeouts).
4. Rebuild the client with fresh kubeconfig if credentials expired.

Example fix

// before
Watch watch = k8sUtils.createBatchJobWatcher(jobName, watcher);
// after
Watch watch = null;
try {
    watch = k8sUtils.createBatchJobWatcher(jobName, watcher);
} catch (TaskException e) {
    log.warn("watch registration failed, falling back to polling", e);
    pollJobStatus(jobName, namespace);
}

if (!apiServerReachable()) {
    throw new IllegalStateException("cannot register watch: API server unreachable");
}

boolean hasWatchPermission(KubernetesClient client, String ns) {
    try { client.batch().v1().jobs().inNamespace(ns).withName("__probe__").watch(l -> {}).close(); return true; }
    catch (Exception e) { return false; }
}

try {
    watch = k8sUtils.createBatchJobWatcher(jobName, watcher);
} catch (TaskException e) {
    log.warn("watch failed, falling back to polling", e);
    startPolling(jobName, namespace, watcher);
}

When it happens

Trigger: Calling createBatchJobWatcher(jobName, watcher) when the watch connection cannot be established: API server unreachable, kubeconfig invalid, missing watch permission on batch/jobs, or the resource watch endpoint rejected the request.

Common situations: K8s task log collection failing on clusters with restricted RBAC (no watch verb); proxy/firewall cutting long-lived connections; TLS certificate issues; client built with expired credentials.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• fail to check job:
• fail to create job
• fail to delete job
• Cancel application failed!
• CLUSTER_NOT_EXISTS

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/da7c097d66816175. Report an issue: GitHub.

Appendix: source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/K8sUtils.java:82

    public Boolean jobExist(String jobName, String namespace) {
        try {
            Job job = client.batch().v1().jobs().inNamespace(namespace).withName(jobName).get();
            return job != null;
        } catch (Exception e) {
            throw new TaskException("fail to check job: ", e);
        }
    }

    public Watch createBatchJobWatcher(String jobName, Watcher<Job> watcher) {
        try {
            return client.batch()
                    .v1()
                    .jobs()
                    .withName(jobName)
                    .watch(watcher);
        } catch (Exception e) {
            throw new TaskException("fail to register batch job watcher", e);
        }
    }

    public String getPodLog(String jobName, String namespace) {
        try {
            List<Pod> podList = client.pods().inNamespace(namespace).list().getItems();
            String podName = null;
            for (Pod pod : podList) {
                podName = pod.getMetadata().getName();
                if (podName.contains("-") && jobName.equals(podName.substring(0, podName.lastIndexOf("-")))) {
                    break;
                }
            }
            return client.pods().inNamespace(namespace)
                    .withName(podName)
                    .tailingLines(LOG_LINES)
                    .getLog(Boolean.TRUE);
        } catch (Exception e) {

View on GitHub (pinned to 02eac45a1b)