apache/seatunnel · error · SeaTunnelEngineException

Job id %s restore failed, can not get job state

Error message

Job id %s restore failed, can not get job state

What it means

restoreJobFromMasterActiveSwitch() re-reads a job's state from runningJobStateIMap (with retries) to decide whether to restore, clean up, or drop it. If the IMap read keeps failing past the retry budget, a SeaTunnelEngineException 'Job id %s restore failed, can not get job state' wrapping the cause is thrown, and that job cannot be restored on the new master.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java:1118

     * restart mode and re-enqueued as {@link PendingSourceState#RESTORE}, allowing the job to reuse
     * the standard pending-job scheduling flow on the new master.
     *
     * @param jobId restored job identifier
     * @param jobInfo distributed immutable job metadata captured before the master switch
     */
    private void restoreJobFromMasterActiveSwitch(@NonNull Long jobId, @NonNull JobInfo jobInfo) {
        Object jobState;
        try {
            jobState =
                    RetryUtils.retryWithException(
                            () -> runningJobStateIMap.get(jobId),
                            new RetryUtils.RetryMaterial(
                                    Constant.OPERATION_RETRY_TIME,
                                    true,
                                    ExceptionUtil::isOperationNeedRetryException,
                                    Constant.OPERATION_RETRY_SLEEP));
        } catch (Exception e) {
            throw new SeaTunnelEngineException(
                    String.format("Job id %s restore failed, can not get job state", jobId), e);
        }
        if (jobState == null) {
            runningJobInfoIMap.remove(jobId);
            return;
        }
        if (jobState instanceof JobStatus && ((JobStatus) jobState).isEndState()) {
            JobCleanupRecord cleanupRecord =
                    pendingJobCleanupIMap != null ? pendingJobCleanupIMap.get(jobId) : null;
            if (cleanupRecord != null) {
                schedulePendingJobCleanup(jobId, cleanupRecord);
            } else {
                cleanupTerminalZombieJob(jobId, jobInfo, (JobStatus) jobState);
            }
            return;
        }

        JobMaster jobMaster =

View on GitHub (pinned to cf67b549a7)

Solutions

1. Check the wrapped exception for Hazelcast operation-timeout causes and stabilize cluster connectivity
2. Retry the master activation / restart the node so the restore pass reruns
3. Increase OPERATION_RETRY_TIME or the retry predicate coverage if cluster formation is slow
4. Verify runningJobStateIMap backup counts and partition health; consider tuning Hazelcast backup/timeout settings
5. If the job is truly orphaned, manually clean its IMap entries (runningJobInfoIMap/runningJobStateIMap) for the jobId and resubmit the job

Example fix

// before
}catch (Exception e) {
    throw new SeaTunnelEngineException(
            String.format("Job id %s restore failed, can not get job state", jobId), e);
}
// after
}catch (Exception e) {
    // log-and-continue for this job, surfaced via executor catch
    logger.severe("Cannot get state for job " + jobId + ": " + ExceptionUtils.getMessage(e));
    throw new SeaTunnelEngineException(
            String.format("Job id %s restore failed, can not get job state", jobId), e);
} // address root cause: Hazelcast map availability during failover

// before restoring, probe job state availability
Object state = runningJobStateIMap.get(jobId); // wrapped in your own retry
if (state == null) {
    logger.warning("Job " + jobId + " state missing from IMap; clean stale entries or resubmit");
}

try {
    restoreJob(jobId, jobInfo);
} catch (SeaTunnelEngineException e
        && e.getMessage().endsWith("can not get job state")) {
    logger.warning("State IMap unreadable for job " + jobId + ", cause: " + e.getCause());
    // retry after cluster stabilizes or clean IMap entries and resubmit
}

When it happens

Trigger: Per-job restore after master failover when runningJobStateIMap.get(jobId) repeatedly throws (Hazelcast operation timeout, instance not active, partition not ready) exceeding Constant.OPERATION_RETRY_TIME retries.

Common situations: Failover while Hazelcast partitions for the job state map are still migrating; unstable network to backup replicas; very large clusters with slow rebalancing after the old master died.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Failed to fetch running jobs from IMap during master switch
• Failed to load readyToCloseStartingTask from IMap, key: %s
• Failed to persist readyToCloseStartingTask to IMap, key: %s
• Job id %s restore interrupted while entering pending queue
• wait worker register error

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/f5451b57caa1aab8. Report an issue: GitHub.