{"record":{"id":"021a3f1928b19a3d","repo":"apache/seatunnel","slug":"job-id-s-restore-interrupted-while-entering-pendi","errorCode":null,"errorMessage":"Job id %s restore interrupted while entering pending queue","messagePattern":"Job id (.+?) restore interrupted while entering pending queue","errorType":"exception","errorClass":"SeaTunnelEngineException","httpStatus":null,"severity":"warning","filePath":"seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java","lineNumber":1162,"sourceCode":"                        runningJobStateIMap,\n                        runningJobStateTimestampsIMap,\n                        ownedSlotProfilesIMap,\n                        runningJobInfoIMap,\n                        engineConfig,\n                        seaTunnelServer);\n\n        try {\n            jobMaster.init(jobInfo.getInitializationTimestamp(), true);\n        } catch (Exception e) {\n            throw new SeaTunnelEngineException(String.format(\"Job id %s init failed\", jobId), e);\n        }\n\n        PendingJobInfo pendingJobInfo = new PendingJobInfo(PendingSourceState.RESTORE, jobMaster);\n        try {\n            pendingJobQueue.put(pendingJobInfo);\n        } catch (InterruptedException e) {\n            Thread.currentThread().interrupt();\n            throw new SeaTunnelEngineException(\n                    String.format(\n                            \"Job id %s restore interrupted while entering pending queue\", jobId),\n                    e);\n        }\n        jobMaster.getPhysicalPlan().updateJobState(JobStatus.PENDING);\n        logger.info(String.format(\"The restore job enter pending queue, JobId: %s\", jobId));\n    }\n\n    private void cleanupTerminalZombieJob(long jobId, JobInfo jobInfo, JobStatus finalStatus) {\n        JobImmutableInformation jobImmutableInformation = restoreJobImmutableInformation(jobInfo);\n        cleanupTerminalZombieCheckpointIfNecessary(jobId, jobImmutableInformation, finalStatus);\n        persistTerminalZombieHistoryIfNecessary(jobId, jobImmutableInformation, finalStatus);\n        cleanupPendingJobStateMaps(createTerminalZombieCleanupRecord(jobId, jobInfo, finalStatus));\n        runningJobInfoIMap.remove(jobId);\n    }\n\n    private void cleanupPendingJobStateForRestore(long jobId, JobCleanupRecord record) {\n        removeKeys(runningJobStateIMap, record.getStateKeys());","sourceCodeStart":1144,"sourceCodeEnd":1180,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java#L1144-L1180","documentation":"After successful JobMaster init during restore, the job is enqueued into pendingJobQueue (ArrayBlockingQueue-style put). If the coordinator thread is interrupted while blocked on put (queue full or thread being shut down), the code restores the interrupt flag and throws SeaTunnelEngineException 'Job id %s restore interrupted while entering pending queue'.","triggerScenarios":"pendingJobQueue is full (many jobs restoring at once, pending-job consumer slow) so put() blocks, and the restore thread is interrupted by node shutdown, Hazelcast member teardown, or engine stop; alternatively a direct interrupt from the scheduling executor.","commonSituations":"Mass failover restoring hundreds of jobs simultaneously onto one master while it is being shut down; ThreadPoolShutdown during rolling restart; jobMaster scheduling thread lifecycle racing with restore.","solutions":["Re-run restore by restarting/keeping the master node alive until all restored jobs are enqueued","Avoid shutting down the node while the restore pass is in flight; gate shutdowns on restore completion","Increase pendingJobQueue capacity or speed up the pending-job consumer if the queue frequently fills","Reduce restored-job concurrency or stagger node restarts to avoid backpressure","Check logs to confirm thread.interrupt() source (shutdown hook) and adjust orchestration order"],"exampleFix":"// before\n} catch (InterruptedException e) {\n    Thread.currentThread().interrupt();\n    throw new SeaTunnelEngineException(\n            String.format(\"Job id %s restore interrupted while entering pending queue\", jobId), e);\n}\n// after\n} catch (InterruptedException e) {\n    Thread.currentThread().interrupt(); // preserve interrupt status\n    throw new SeaTunnelEngineException(\n            String.format(\"Job id %s restore interrupted while entering pending queue\", jobId), e);\n} // prevention: keep node alive until restore completes; consider offer-with-timeout instead of put","handlingStrategy":"try-catch","validationCode":"// before shutdown, verify no restore in flight and queue is drained\nif (!pendingJobQueue.isEmpty() || restoreInProgress) {\n    logger.warning(\"Restore in progress; defer node shutdown until pending jobs are enqueued\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    restoreJob(jobId, jobInfo);\n} catch (SeaTunnelEngineException e\n        && e.getMessage().contains(\"restore interrupted while entering pending queue\")) {\n    Thread.currentThread().interrupt(); // keep interrupt status\n    logger.warning(\"Shutdown raced job restore for \" + jobId + \"; job will re-restore on next master\");\n}","preventionTips":["Gate node shutdown on restore/pending-queue completion","Size pendingJobQueue for worst-case simultaneous restorations after failover","Restart nodes one at a time and wait for PENDING jobs to resume","Monitor pendingJobQueue depth and consumer throughput"],"tags":["interrupted","pending-queue","failover","job-restore","zeta-engine"],"backgroundTag":"invalid-state-transition","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}