{"record":{"id":"1a2cb33272785526","repo":"apache/seatunnel","slug":"failed-to-fetch-running-jobs-from-imap-during-mast","errorCode":null,"errorMessage":"Failed to fetch running jobs from IMap during master switch restore","messagePattern":"Failed to fetch running jobs from IMap during master switch restore","errorType":"exception","errorClass":"SeaTunnelEngineException","httpStatus":null,"severity":"error","filePath":"seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java","lineNumber":1007,"sourceCode":"    private void restoreAllRunningJobFromMasterNodeSwitch() {\n        List<Map.Entry<Long, JobInfo>> needRestoreFromMasterNodeSwitchJobs;\n        try {\n            needRestoreFromMasterNodeSwitchJobs =\n                    RetryUtils.retryWithException(\n                            () ->\n                                    runningJobInfoIMap.entrySet().stream()\n                                            .filter(\n                                                    entry ->\n                                                            !runningJobMasterMap.containsKey(\n                                                                    entry.getKey()))\n                                            .collect(Collectors.toList()),\n                            new RetryUtils.RetryMaterial(\n                                    Constant.OPERATION_RETRY_TIME,\n                                    true,\n                                    ExceptionUtil::isOperationNeedRetryException,\n                                    Constant.OPERATION_RETRY_SLEEP));\n        } catch (Exception e) {\n            throw new SeaTunnelEngineException(\n                    \"Failed to fetch running jobs from IMap during master switch restore\", e);\n        }\n        if (needRestoreFromMasterNodeSwitchJobs.isEmpty()) {\n            return;\n        }\n        // Pre-filter: clean up terminal-state zombie jobs immediately before waiting for workers.\n        // Zombies do not need a worker — they only need IMap cleanup. Processing them here avoids\n        // blocking zombie cleanup behind the worker-wait loop.\n        Iterator<Map.Entry<Long, JobInfo>> zombieIterator =\n                needRestoreFromMasterNodeSwitchJobs.iterator();\n        while (zombieIterator.hasNext()) {\n            Map.Entry<Long, JobInfo> entry = zombieIterator.next();\n            Object jobState;\n            try {\n                jobState =\n                        RetryUtils.retryWithException(\n                                () -> runningJobStateIMap.get(entry.getKey()),\n                                new RetryUtils.RetryMaterial(","sourceCodeStart":989,"sourceCodeEnd":1025,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java#L989-L1025","documentation":"During failover, when this node becomes the active master, CoordinatorService.restoreAllRunningJobFromMasterNodeSwitch() scans runningJobInfoIMap for jobs that no longer have a local JobMaster so they can be restored. If reading the IMap keeps failing after exhausting the retry budget (Constant.OPERATION_RETRY_TIME with isOperationNeedRetryException), a SeaTunnelEngineException wrapping the cause is thrown and master activation fails.","triggerScenarios":"Master node crash/failover while Hazelcast IMap partitions are unavailable or migrating; network partition between the new master and remaining members; IMap backend timeouts exceeding the configured retry count during cluster re-formation.","commonSituations":"Split-brain or rolling restart of the Zeta cluster where the new master starts before Hazelcast distributed maps are consistent; severe GC pauses or network flapping in the cluster; misconfigured Hazelcast networking (e.g. TCP-IP member list pointing at unreachable hosts).","solutions":["Inspect the wrapped cause (getCause()) for Hazelcast HazelcastInstanceNotActiveException/OperationTimeoutException and fix the underlying cluster connectivity issue","Restart the failed master node so it rejoins the cluster and retries restore","Increase Constant.OPERATION_RETRY_TIME / OPERATION_RETRY_SLEEP to tolerate slower cluster formation","Verify Hazelcast cluster state (all members reachable, partition migration complete) before starting the master","Check network/firewall between cluster members; ensure the member list in config matches actual hosts"],"exampleFix":"// before\n}catch (Exception e) {\n    throw new SeaTunnelEngineException(\n            \"Failed to fetch running jobs from IMap during master switch restore\", e);\n}\n// after\n}catch (Exception e) {\n    logger.severe(\"IMap fetch failed during master switch restore: \" + ExceptionUtils.getMessage(e));\n    throw new SeaTunnelEngineException(\n            \"Failed to fetch running jobs from IMap during master switch restore\", e);\n} // fix root cause: ensure Hazelcast members are reachable and maps are populated","handlingStrategy":"retry","validationCode":"// before starting/activating a master node, verify Hazelcast cluster health\nHazelcastInstance hz = ...;\nif (!hz.getCluster().getMembers().isEmpty()\n        && hz.getLifecycleService().isRunning()) {\n    logger.info(\"Cluster healthy; safe to activate master and run restore\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    coordinatorRestore();\n} catch (SeaTunnelEngineException e\n        && e.getMessage().contains(\"Failed to fetch running jobs from IMap\")) {\n    logger.warning(\"Cluster not ready for master restore, will retry: \" + e.getCause());\n    // backoff and restart the node or retry activation\n}","preventionTips":["Keep Hazelcast member lists and networking config correct and reachable","Allow partition migration to settle before electing/activating a master","Tune OPERATION_RETRY_TIME/OPERATION_RETRY_SLEEP for slow cluster formation","Monitor cluster connectivity and GC pauses that stall Hazelcast operations"],"tags":["hazelcast","imap","failover","master-switch","zeta-engine"],"backgroundTag":"http-request-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}