apache/seatunnel · warning · SeaTunnelEngineException

Job id %s restore interrupted while entering pending queue

Error message

Job id %s restore interrupted while entering pending queue

What it means

After successful JobMaster init during restore, the job is enqueued into pendingJobQueue (ArrayBlockingQueue-style put). If the coordinator thread is interrupted while blocked on put (queue full or thread being shut down), the code restores the interrupt flag and throws SeaTunnelEngineException 'Job id %s restore interrupted while entering pending queue'.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java:1162

                        runningJobStateIMap,
                        runningJobStateTimestampsIMap,
                        ownedSlotProfilesIMap,
                        runningJobInfoIMap,
                        engineConfig,
                        seaTunnelServer);

        try {
            jobMaster.init(jobInfo.getInitializationTimestamp(), true);
        } catch (Exception e) {
            throw new SeaTunnelEngineException(String.format("Job id %s init failed", jobId), e);
        }

        PendingJobInfo pendingJobInfo = new PendingJobInfo(PendingSourceState.RESTORE, jobMaster);
        try {
            pendingJobQueue.put(pendingJobInfo);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new SeaTunnelEngineException(
                    String.format(
                            "Job id %s restore interrupted while entering pending queue", jobId),
                    e);
        }
        jobMaster.getPhysicalPlan().updateJobState(JobStatus.PENDING);
        logger.info(String.format("The restore job enter pending queue, JobId: %s", jobId));
    }

    private void cleanupTerminalZombieJob(long jobId, JobInfo jobInfo, JobStatus finalStatus) {
        JobImmutableInformation jobImmutableInformation = restoreJobImmutableInformation(jobInfo);
        cleanupTerminalZombieCheckpointIfNecessary(jobId, jobImmutableInformation, finalStatus);
        persistTerminalZombieHistoryIfNecessary(jobId, jobImmutableInformation, finalStatus);
        cleanupPendingJobStateMaps(createTerminalZombieCleanupRecord(jobId, jobInfo, finalStatus));
        runningJobInfoIMap.remove(jobId);
    }

    private void cleanupPendingJobStateForRestore(long jobId, JobCleanupRecord record) {
        removeKeys(runningJobStateIMap, record.getStateKeys());

View on GitHub (pinned to cf67b549a7)

Solutions

Re-run restore by restarting/keeping the master node alive until all restored jobs are enqueued
Avoid shutting down the node while the restore pass is in flight; gate shutdowns on restore completion
Increase pendingJobQueue capacity or speed up the pending-job consumer if the queue frequently fills
Reduce restored-job concurrency or stagger node restarts to avoid backpressure
Check logs to confirm thread.interrupt() source (shutdown hook) and adjust orchestration order

Example fix

// before
} catch (InterruptedException e) {
    Thread.currentThread().interrupt();
    throw new SeaTunnelEngineException(
            String.format("Job id %s restore interrupted while entering pending queue", jobId), e);
}
// after
} catch (InterruptedException e) {
    Thread.currentThread().interrupt(); // preserve interrupt status
    throw new SeaTunnelEngineException(
            String.format("Job id %s restore interrupted while entering pending queue", jobId), e);
} // prevention: keep node alive until restore completes; consider offer-with-timeout instead of put

Defensive patterns

Strategy: try-catch

Validate before calling

// before shutdown, verify no restore in flight and queue is drained
if (!pendingJobQueue.isEmpty() || restoreInProgress) {
    logger.warning("Restore in progress; defer node shutdown until pending jobs are enqueued");
}

Try / catch

try {
    restoreJob(jobId, jobInfo);
} catch (SeaTunnelEngineException e
        && e.getMessage().contains("restore interrupted while entering pending queue")) {
    Thread.currentThread().interrupt(); // keep interrupt status
    logger.warning("Shutdown raced job restore for " + jobId + "; job will re-restore on next master");
}

Prevention

Gate node shutdown on restore/pending-queue completion
Size pendingJobQueue for worst-case simultaneous restorations after failover
Restart nodes one at a time and wait for PENDING jobs to resume
Monitor pendingJobQueue depth and consumer throughput

When it happens

Trigger: pendingJobQueue is full (many jobs restoring at once, pending-job consumer slow) so put() blocks, and the restore thread is interrupted by node shutdown, Hazelcast member teardown, or engine stop; alternatively a direct interrupt from the scheduling executor.

Common situations: Mass failover restoring hundreds of jobs simultaneously onto one master while it is being shut down; ThreadPoolShutdown during rolling restart; jobMaster scheduling thread lifecycle racing with restore.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/021a3f1928b19a3d. Report an issue: GitHub.