apache/seatunnel · error · SeaTunnelEngineException

Job id restore failed, can not get job state

Error message

Job id %s restore failed, can not get job state

What it means

restoreJobFromMasterActiveSwitch() re-reads a job's state from runningJobStateIMap (with retries) to decide whether to restore, clean up, or drop it. If the IMap read keeps failing past the retry budget, a SeaTunnelEngineException 'Job id %s restore failed, can not get job state' wrapping the cause is thrown, and that job cannot be restored on the new master.

Solutions

1. Check the wrapped exception for Hazelcast operation-timeout causes and stabilize cluster connectivity
2. Retry the master activation / restart the node so the restore pass reruns
3. Increase OPERATION_RETRY_TIME or the retry predicate coverage if cluster formation is slow
4. Verify runningJobStateIMap backup counts and partition health; consider tuning Hazelcast backup/timeout settings
5. If the job is truly orphaned, manually clean its IMap entries (runningJobInfoIMap/runningJobStateIMap) for the jobId and resubmit the job

Example fix

// before
}catch (Exception e) {
    throw new SeaTunnelEngineException(
            String.format("Job id %s restore failed, can not get job state", jobId), e);
}
// after
}catch (Exception e) {
    // log-and-continue for this job, surfaced via executor catch
    logger.severe("Cannot get state for job " + jobId + ": " + ExceptionUtils.getMessage(e));
    throw new SeaTunnelEngineException(
            String.format("Job id %s restore failed, can not get job state", jobId), e);
} // address root cause: Hazelcast map availability during failover

// before restoring, probe job state availability
Object state = runningJobStateIMap.get(jobId); // wrapped in your own retry
if (state == null) {
    logger.warning("Job " + jobId + " state missing from IMap; clean stale entries or resubmit");
}

try {
    restoreJob(jobId, jobInfo);
} catch (SeaTunnelEngineException e
        && e.getMessage().endsWith("can not get job state")) {
    logger.warning("State IMap unreadable for job " + jobId + ", cause: " + e.getCause());
    // retry after cluster stabilizes or clean IMap entries and resubmit
}

When it happens

Trigger: Per-job restore after master failover when runningJobStateIMap.get(jobId) repeatedly throws (Hazelcast operation timeout, instance not active, partition not ready) exceeding Constant.OPERATION_RETRY_TIME retries.

Common situations: Failover while Hazelcast partitions for the job state map are still migrating; unstable network to backup replicas; very large clusters with slow rebalancing after the old master died.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Failed to fetch running jobs from IMap during master switch…
• Failed to load readyToCloseStartingTask from IMap, key
• Failed to persist readyToCloseStartingTask to IMap, key
• Job id restore interrupted while entering pending queue
• Can not get coordinator service from an active master node.

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/f5451b57caa1aab8. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java:1118

     * restart mode and re-enqueued as {@link PendingSourceState#RESTORE}, allowing the job to reuse
     * the standard pending-job scheduling flow on the new master.
     *
     * @param jobId restored job identifier
     * @param jobInfo distributed immutable job metadata captured before the master switch
     */
    private void restoreJobFromMasterActiveSwitch(@NonNull Long jobId, @NonNull JobInfo jobInfo) {
        Object jobState;
        try {
            jobState =
                    RetryUtils.retryWithException(
                            () -> runningJobStateIMap.get(jobId),
                            new RetryUtils.RetryMaterial(
                                    Constant.OPERATION_RETRY_TIME,
                                    true,
                                    ExceptionUtil::isOperationNeedRetryException,
                                    Constant.OPERATION_RETRY_SLEEP));
        } catch (Exception e) {
            throw new SeaTunnelEngineException(
                    String.format("Job id %s restore failed, can not get job state", jobId), e);
        }
        if (jobState == null) {
            runningJobInfoIMap.remove(jobId);
            return;
        }
        if (jobState instanceof JobStatus && ((JobStatus) jobState).isEndState()) {
            JobCleanupRecord cleanupRecord =
                    pendingJobCleanupIMap != null ? pendingJobCleanupIMap.get(jobId) : null;
            if (cleanupRecord != null) {
                schedulePendingJobCleanup(jobId, cleanupRecord);
            } else {
                cleanupTerminalZombieJob(jobId, jobInfo, (JobStatus) jobState);
            }
            return;
        }

        JobMaster jobMaster =

View on GitHub (pinned to cf67b549a7)