apache/seatunnel · error · SeaTunnelEngineException

Failed to fetch running jobs from IMap during master switch

Error message

Failed to fetch running jobs from IMap during master switch restore

What it means

During failover, when this node becomes the active master, CoordinatorService.restoreAllRunningJobFromMasterNodeSwitch() scans runningJobInfoIMap for jobs that no longer have a local JobMaster so they can be restored. If reading the IMap keeps failing after exhausting the retry budget (Constant.OPERATION_RETRY_TIME with isOperationNeedRetryException), a SeaTunnelEngineException wrapping the cause is thrown and master activation fails.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java:1007

    private void restoreAllRunningJobFromMasterNodeSwitch() {
        List<Map.Entry<Long, JobInfo>> needRestoreFromMasterNodeSwitchJobs;
        try {
            needRestoreFromMasterNodeSwitchJobs =
                    RetryUtils.retryWithException(
                            () ->
                                    runningJobInfoIMap.entrySet().stream()
                                            .filter(
                                                    entry ->
                                                            !runningJobMasterMap.containsKey(
                                                                    entry.getKey()))
                                            .collect(Collectors.toList()),
                            new RetryUtils.RetryMaterial(
                                    Constant.OPERATION_RETRY_TIME,
                                    true,
                                    ExceptionUtil::isOperationNeedRetryException,
                                    Constant.OPERATION_RETRY_SLEEP));
        } catch (Exception e) {
            throw new SeaTunnelEngineException(
                    "Failed to fetch running jobs from IMap during master switch restore", e);
        }
        if (needRestoreFromMasterNodeSwitchJobs.isEmpty()) {
            return;
        }
        // Pre-filter: clean up terminal-state zombie jobs immediately before waiting for workers.
        // Zombies do not need a worker — they only need IMap cleanup. Processing them here avoids
        // blocking zombie cleanup behind the worker-wait loop.
        Iterator<Map.Entry<Long, JobInfo>> zombieIterator =
                needRestoreFromMasterNodeSwitchJobs.iterator();
        while (zombieIterator.hasNext()) {
            Map.Entry<Long, JobInfo> entry = zombieIterator.next();
            Object jobState;
            try {
                jobState =
                        RetryUtils.retryWithException(
                                () -> runningJobStateIMap.get(entry.getKey()),
                                new RetryUtils.RetryMaterial(

View on GitHub (pinned to cf67b549a7)

Solutions

Inspect the wrapped cause (getCause()) for Hazelcast HazelcastInstanceNotActiveException/OperationTimeoutException and fix the underlying cluster connectivity issue
Restart the failed master node so it rejoins the cluster and retries restore
Increase Constant.OPERATION_RETRY_TIME / OPERATION_RETRY_SLEEP to tolerate slower cluster formation
Verify Hazelcast cluster state (all members reachable, partition migration complete) before starting the master
Check network/firewall between cluster members; ensure the member list in config matches actual hosts

Example fix

// before
}catch (Exception e) {
    throw new SeaTunnelEngineException(
            "Failed to fetch running jobs from IMap during master switch restore", e);
}
// after
}catch (Exception e) {
    logger.severe("IMap fetch failed during master switch restore: " + ExceptionUtils.getMessage(e));
    throw new SeaTunnelEngineException(
            "Failed to fetch running jobs from IMap during master switch restore", e);
} // fix root cause: ensure Hazelcast members are reachable and maps are populated

Defensive patterns

Strategy: retry

Validate before calling

// before starting/activating a master node, verify Hazelcast cluster health
HazelcastInstance hz = ...;
if (!hz.getCluster().getMembers().isEmpty()
        && hz.getLifecycleService().isRunning()) {
    logger.info("Cluster healthy; safe to activate master and run restore");
}

Try / catch

try {
    coordinatorRestore();
} catch (SeaTunnelEngineException e
        && e.getMessage().contains("Failed to fetch running jobs from IMap")) {
    logger.warning("Cluster not ready for master restore, will retry: " + e.getCause());
    // backoff and restart the node or retry activation
}

Prevention

Keep Hazelcast member lists and networking config correct and reachable
Allow partition migration to settle before electing/activating a master
Tune OPERATION_RETRY_TIME/OPERATION_RETRY_SLEEP for slow cluster formation
Monitor cluster connectivity and GC pauses that stall Hazelcast operations

When it happens

Trigger: Master node crash/failover while Hazelcast IMap partitions are unavailable or migrating; network partition between the new master and remaining members; IMap backend timeouts exceeding the configured retry count during cluster re-formation.

Common situations: Split-brain or rolling restart of the Zeta cluster where the new master starts before Hazelcast distributed maps are consistent; severe GC pauses or network flapping in the cluster; misconfigured Hazelcast networking (e.g. TCP-IP member list pointing at unreachable hosts).

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/1a2cb33272785526. Report an issue: GitHub.