apache/seatunnel · warning · SeaTunnelEngineRetryableException

Can not get coordinator service from an active master node.

Error message

Can not get coordinator service from an active master node.

What it means

SeaTunnelServer.getCoordinatorService throws SeaTunnelEngineRetryableException when the node IS the master but its CoordinatorService is not active yet (coordinator still initializing after becoming master). The comment in the source states this is intentional: throwing a retryable exception releases the operation thread so the worker can retry the operation later.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/SeaTunnelServer.java:313

                try {
                    LOGGER.warning(
                            "This is master node, waiting the coordinator service init finished");
                    Thread.sleep(retryPause);
                    retryCount++;
                } catch (InterruptedException e) {
                    throw new RuntimeException(e);
                }
            }
            if (coordinatorService.isCoordinatorActive()) {
                return coordinatorService;
            }

            if (!isMasterNode()) {
                throw new SeaTunnelEngineException("This is not a master node now.");
            }
            // Return retryable exception to retry from the worker node, because the coordinator is
            // not ready yet. By this way, we can release the operation thread and retry later.
            throw new SeaTunnelEngineRetryableException(
                    "Can not get coordinator service from an active master node.");
        } else {
            throw new SeaTunnelEngineException(
                    "Please don't get coordinator service from an inactive master node");
        }
    }

    public RealtimeMetricsService getRealtimeMetricsService() {
        return realtimeMetricsService;
    }

    synchronized void startRealtimeMetricsService(CoordinatorService activeCoordinatorService) {
        if (realtimeMetricsService != null) {
            return;
        }
        realtimeMetricsService =
                new RealtimeMetricsService((NodeEngineImpl) nodeEngine, activeCoordinatorService);
        realtimeMetricsService.start();

View on GitHub (pinned to cf67b549a7)

Solutions

No manual fix needed: it's a retryable exception and the worker's operation retry loop will re-execute once the coordinator becomes active.
If retries exhaust, increase the operation retry budget/check interval on workers or shorten coordinator startup time (reduce job restore load).
Verify master election works properly — check Hazelcast logs if the coordinator never becomes active.

Defensive patterns

Strategy: retry

Try / catch

catch (SeaTunnelEngineRetryableException e) {
    // coordinator not ready yet on master — safe to retry
    Uninterruptibles.sleepUninterruptibly(Constant.OPERATION_RETRY_SLEEP, TimeUnit.MILLISECONDS);
    return retryOp();
}

Prevention

This is expected right after master election — always wrap coordinator calls in retry logic
Give the new master time to activate its coordinator before submitting jobs
Reduce job restore workload to shorten the coordinator activation window

When it happens

Trigger: Worker operations (task deployment, state updates, resource manager calls) arriving at the newly elected master while it is still initializing its CoordinatorService after failover or cluster startup.

Common situations: Master node crash followed by election; large cluster where coordinator activation (restoring jobs) takes seconds; burst of worker operations hitting the master immediately after it takes over.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/fabbd2f0a21a6d75. Report an issue: GitHub.