apache/seatunnel · error · SeaTunnelEngineException

wait worker register error

Error message

wait worker register error

What it means

After a master switch, restoreAllRunningJobFromMasterNodeSwitch() blocks until at least one worker registers with the ResourceManager, polling every second. The wait loop is interrupted via Thread.currentThread() interrupt; the code logs it and throws SeaTunnelEngineException('wait worker register error'). This means the coordinator thread waiting for workers was interrupted rather than workers being unavailable per se.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java:1053

                                entry.getKey(), e.getMessage()));
                continue;
            }
            if (jobState instanceof JobStatus && ((JobStatus) jobState).isEndState()) {
                restoreJobFromMasterActiveSwitch(entry.getKey(), entry.getValue());
                zombieIterator.remove();
            }
        }
        if (needRestoreFromMasterNodeSwitchJobs.isEmpty()) {
            return;
        }
        // waiting have worker registered
        while (getResourceManager().workerCount(Collections.emptyMap()) == 0) {
            try {
                logger.info("Waiting for worker registered");
                Thread.sleep(1000);
            } catch (InterruptedException e) {
                logger.severe(ExceptionUtils.getMessage(e));
                throw new SeaTunnelEngineException("wait worker register error", e);
            }
        }
        List<CompletableFuture<Void>> collect =
                needRestoreFromMasterNodeSwitchJobs.stream()
                        .map(
                                entry ->
                                        CompletableFuture.runAsync(
                                                () -> {
                                                    logger.info(
                                                            String.format(
                                                                    "begin restore job (%s) from master active switch",
                                                                    entry.getKey()));
                                                    try {
                                                        // skip the job new submit
                                                        if (!runningJobMasterMap.containsKey(
                                                                entry.getKey())) {
                                                            restoreJobFromMasterActiveSwitch(
                                                                    entry.getKey(),

View on GitHub (pinned to cf67b549a7)

Solutions

1. Ensure at least one worker node is started and registered before/while masters activate
2. Avoid shutting down or interrupting the master during the worker-wait window; stop workers after jobs are restored
3. Check logs for what interrupted the thread (shutdown hook, Hazelcast member removal) and sequence node lifecycle accordingly
4. Increase cluster startup orchestration delays so workers register promptly after master election
5. If restore is failing during scheduled restarts, move job restore off the shutdown path or increase OPERATION_RETRY settings

Example fix

// before
while (getResourceManager().workerCount(Collections.emptyMap()) == 0) {
    Thread.sleep(1000);
}
// after
long deadline = System.currentTimeMillis() + 120_000L;
while (getResourceManager().workerCount(Collections.emptyMap()) == 0) {
    if (System.currentTimeMillis() > deadline) {
        throw new SeaTunnelEngineException("Timed out waiting for worker registration");
    }
    Thread.sleep(1000);
}

// before triggering master restore, ensure workers are registered
if (resourceManager.workerCount(Collections.emptyMap()) == 0) {
    logger.warning("No workers registered; start workers before master activation/restore");
}

try {
    restoreAllRunningJobs();
} catch (SeaTunnelEngineException e
        && e.getMessage().equals("wait worker register error")) {
    logger.warning("Restore wait loop interrupted; check node shutdown ordering: " + e.getCause());
}

When it happens

Trigger: The coordinator/master thread executing restoreAllRunningJobFromMasterNodeSwitch receives an interrupt (node shutdown, thread pool termination, engine stop) while sleeping in the 1-second poll loop waiting for worker count > 0.

Common situations: Shutting down a Zeta master node while no workers have registered yet (e.g. cluster startup racing with shutdown); killing the node process; deployment scripts that restart masters before workers join.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Job id %s restore interrupted while entering pending queue
• Failed to fetch running jobs from IMap during master switch
• Job id %s restore failed, can not get job state
• Job id %s init failed
• Job %s not running (restore in progress)

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/64a7942a464ba841. Report an issue: GitHub.