apache/dolphinscheduler · error · TaskException

fail to register batch job watcher

Error message

fail to register batch job watcher

What it means

K8sUtils.createBatchJobWatcher() registers a Watch on a named Batch/V1 Job and wraps any failure in a TaskException 'fail to register batch job watcher'. Watch registration is an HTTP long-poll against the API server, so network issues and RBAC are the typical root causes.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/K8sUtils.java:82

    public Boolean jobExist(String jobName, String namespace) {
        try {
            Job job = client.batch().v1().jobs().inNamespace(namespace).withName(jobName).get();
            return job != null;
        } catch (Exception e) {
            throw new TaskException("fail to check job: ", e);
        }
    }

    public Watch createBatchJobWatcher(String jobName, Watcher<Job> watcher) {
        try {
            return client.batch()
                    .v1()
                    .jobs()
                    .withName(jobName)
                    .watch(watcher);
        } catch (Exception e) {
            throw new TaskException("fail to register batch job watcher", e);
        }
    }

    public String getPodLog(String jobName, String namespace) {
        try {
            List<Pod> podList = client.pods().inNamespace(namespace).list().getItems();
            String podName = null;
            for (Pod pod : podList) {
                podName = pod.getMetadata().getName();
                if (podName.contains("-") && jobName.equals(podName.substring(0, podName.lastIndexOf("-")))) {
                    break;
                }
            }
            return client.pods().inNamespace(namespace)
                    .withName(podName)
                    .tailingLines(LOG_LINES)
                    .getLog(Boolean.TRUE);
        } catch (Exception e) {

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Check the chained cause for KubernetesClientException details (403 vs connection error).
2. Verify RBAC includes 'watch' on batch/jobs in the target namespace.
3. Ensure network path allows long-lived HTTPS connections to the API server (no aggressive proxy timeouts).
4. Rebuild the client with fresh kubeconfig if credentials expired.

Example fix

// before
Watch watch = k8sUtils.createBatchJobWatcher(jobName, watcher);
// after
Watch watch = null;
try {
    watch = k8sUtils.createBatchJobWatcher(jobName, watcher);
} catch (TaskException e) {
    log.warn("watch registration failed, falling back to polling", e);
    pollJobStatus(jobName, namespace);
}

if (!apiServerReachable()) {
    throw new IllegalStateException("cannot register watch: API server unreachable");
}

boolean hasWatchPermission(KubernetesClient client, String ns) {
    try { client.batch().v1().jobs().inNamespace(ns).withName("__probe__").watch(l -> {}).close(); return true; }
    catch (Exception e) { return false; }
}

try {
    watch = k8sUtils.createBatchJobWatcher(jobName, watcher);
} catch (TaskException e) {
    log.warn("watch failed, falling back to polling", e);
    startPolling(jobName, namespace, watcher);
}

When it happens

Trigger: Calling createBatchJobWatcher(jobName, watcher) when the watch connection cannot be established: API server unreachable, kubeconfig invalid, missing watch permission on batch/jobs, or the resource watch endpoint rejected the request.

Common situations: K8s task log collection failing on clusters with restricted RBAC (no watch verb); proxy/firewall cutting long-lived connections; TLS certificate issues; client built with expired credentials.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• fail to create job
• fail to delete job
• fail to check job:
• namespace %s does not exist in k8s cluster, please create na
• fail to get k8s ApiClient:%s

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/da7c097d66816175. Report an issue: GitHub.