{"record":{"id":"f5451b57caa1aab8","repo":"apache/seatunnel","slug":"job-id-s-restore-failed-can-not-get-job-state","errorCode":null,"errorMessage":"Job id %s restore failed, can not get job state","messagePattern":"Job id (.+?) restore failed, can not get job state","errorType":"exception","errorClass":"SeaTunnelEngineException","httpStatus":null,"severity":"error","filePath":"seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java","lineNumber":1118,"sourceCode":"     * restart mode and re-enqueued as {@link PendingSourceState#RESTORE}, allowing the job to reuse\n     * the standard pending-job scheduling flow on the new master.\n     *\n     * @param jobId restored job identifier\n     * @param jobInfo distributed immutable job metadata captured before the master switch\n     */\n    private void restoreJobFromMasterActiveSwitch(@NonNull Long jobId, @NonNull JobInfo jobInfo) {\n        Object jobState;\n        try {\n            jobState =\n                    RetryUtils.retryWithException(\n                            () -> runningJobStateIMap.get(jobId),\n                            new RetryUtils.RetryMaterial(\n                                    Constant.OPERATION_RETRY_TIME,\n                                    true,\n                                    ExceptionUtil::isOperationNeedRetryException,\n                                    Constant.OPERATION_RETRY_SLEEP));\n        } catch (Exception e) {\n            throw new SeaTunnelEngineException(\n                    String.format(\"Job id %s restore failed, can not get job state\", jobId), e);\n        }\n        if (jobState == null) {\n            runningJobInfoIMap.remove(jobId);\n            return;\n        }\n        if (jobState instanceof JobStatus && ((JobStatus) jobState).isEndState()) {\n            JobCleanupRecord cleanupRecord =\n                    pendingJobCleanupIMap != null ? pendingJobCleanupIMap.get(jobId) : null;\n            if (cleanupRecord != null) {\n                schedulePendingJobCleanup(jobId, cleanupRecord);\n            } else {\n                cleanupTerminalZombieJob(jobId, jobInfo, (JobStatus) jobState);\n            }\n            return;\n        }\n\n        JobMaster jobMaster =","sourceCodeStart":1100,"sourceCodeEnd":1136,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java#L1100-L1136","documentation":"restoreJobFromMasterActiveSwitch() re-reads a job's state from runningJobStateIMap (with retries) to decide whether to restore, clean up, or drop it. If the IMap read keeps failing past the retry budget, a SeaTunnelEngineException 'Job id %s restore failed, can not get job state' wrapping the cause is thrown, and that job cannot be restored on the new master.","triggerScenarios":"Per-job restore after master failover when runningJobStateIMap.get(jobId) repeatedly throws (Hazelcast operation timeout, instance not active, partition not ready) exceeding Constant.OPERATION_RETRY_TIME retries.","commonSituations":"Failover while Hazelcast partitions for the job state map are still migrating; unstable network to backup replicas; very large clusters with slow rebalancing after the old master died.","solutions":["Check the wrapped exception for Hazelcast operation-timeout causes and stabilize cluster connectivity","Retry the master activation / restart the node so the restore pass reruns","Increase OPERATION_RETRY_TIME or the retry predicate coverage if cluster formation is slow","Verify runningJobStateIMap backup counts and partition health; consider tuning Hazelcast backup/timeout settings","If the job is truly orphaned, manually clean its IMap entries (runningJobInfoIMap/runningJobStateIMap) for the jobId and resubmit the job"],"exampleFix":"// before\n}catch (Exception e) {\n    throw new SeaTunnelEngineException(\n            String.format(\"Job id %s restore failed, can not get job state\", jobId), e);\n}\n// after\n}catch (Exception e) {\n    // log-and-continue for this job, surfaced via executor catch\n    logger.severe(\"Cannot get state for job \" + jobId + \": \" + ExceptionUtils.getMessage(e));\n    throw new SeaTunnelEngineException(\n            String.format(\"Job id %s restore failed, can not get job state\", jobId), e);\n} // address root cause: Hazelcast map availability during failover","handlingStrategy":"retry","validationCode":"// before restoring, probe job state availability\nObject state = runningJobStateIMap.get(jobId); // wrapped in your own retry\nif (state == null) {\n    logger.warning(\"Job \" + jobId + \" state missing from IMap; clean stale entries or resubmit\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    restoreJob(jobId, jobInfo);\n} catch (SeaTunnelEngineException e\n        && e.getMessage().endsWith(\"can not get job state\")) {\n    logger.warning(\"State IMap unreadable for job \" + jobId + \", cause: \" + e.getCause());\n    // retry after cluster stabilizes or clean IMap entries and resubmit\n}","preventionTips":["Tune Hazelcast map backups and operation timeouts for cluster size","Increase retry budget for restore-time IMap reads","Monitor Hazelcast partition migration during failovers","Clean orphaned runningJobInfoIMap/runningJobStateIMap entries after failed restores"],"tags":["hazelcast","imap","failover","job-restore","zeta-engine"],"backgroundTag":"resource-not-found","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}