{"record":{"id":"fabcdb76974624c2","repo":"apache/seatunnel","slug":"the-job-id-s-is-waiting-for-terminal-state-cleanu","errorCode":null,"errorMessage":"The job id %s is waiting for terminal state cleanup, please retry later.","messagePattern":"The job id (.+?) is waiting for terminal state cleanup, please retry later\\.","errorType":"exception","errorClass":"JobException","httpStatus":null,"severity":"error","filePath":"seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java","lineNumber":1419,"sourceCode":"                    JobInfo submittedJobInfo = null;\n                    try {\n                        JobImmutableInformation submittedJobImmutableInformation =\n                                deserializeJobImmutableInformation(jobImmutableInformation);\n                        validateCheckpointRestoreSourceJobIsTerminal(\n                                submittedJobImmutableInformation, jobId);\n                        if (isStartWithSavePoint) {\n                            cleanupPendingPipelineCleanupForRestore(jobId);\n                        }\n                        JobCleanupRecord pendingCleanupRecord =\n                                pendingJobCleanupIMap != null\n                                        ? pendingJobCleanupIMap.get(jobId)\n                                        : null;\n                        if (pendingCleanupRecord != null\n                                && isCleanupOwnedByCurrentJob(jobId, pendingCleanupRecord)) {\n                            if (isStartWithSavePoint) {\n                                cleanupPendingJobStateForRestore(jobId, pendingCleanupRecord);\n                            } else {\n                                throw new JobException(\n                                        String.format(\n                                                \"The job id %s is waiting for terminal state cleanup, please retry later.\",\n                                                jobId));\n                            }\n                        }\n\n                        jobMaster =\n                                new JobMaster(\n                                        jobId,\n                                        jobImmutableInformation,\n                                        this.nodeEngine,\n                                        mdcExecutorService,\n                                        getResourceManager(),\n                                        getJobHistoryService(),\n                                        runningJobStateIMap,\n                                        runningJobStateTimestampsIMap,\n                                        ownedSlotProfilesIMap,\n                                        runningJobInfoIMap,","sourceCodeStart":1401,"sourceCodeEnd":1437,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java#L1401-L1437","documentation":"When a job is submitted with an explicit jobId, the engine checks whether a pending terminal-state cleanup record still owned by that jobId exists in the job state IMap. If one exists and the submission is not a savepoint (restore) start, the submission is rejected with this JobException, telling the caller to retry later. It is a transient rejection while the previous job's end-of-life cleanup finishes.","triggerScenarios":"submitJob with a reused jobId while a pendingCleanupRecord for that id still exists (previous job awaiting cleanup) and isStartWithSavePoint is false.","commonSituations":"Hardcoded jobId resubmitted immediately after stopping the previous job; automated resubmission loops that don't wait for cleanup; fast restart after stop/savepoint with the same fixed id.","solutions":["Wait and retry the submission until the previous job's cleanup completes.","Let the engine generate the jobId (omit the explicit id) so a fresh id is used.","If this is a restore, submit with start-with-savepoint so cleanupPendingJobStateForRestore runs instead of rejecting.","Poll job status/history via CLI or REST to confirm the old job fully terminated before resubmitting."],"exampleFix":"// before: reused fixed id\nlong jobId = 861912345L;\nclient.submitJob(jobId, config, false);\n// after: retry with backoff on a fresh id\nlong jobId = new Random().nextLong();\nclient.submitJob(jobId, config, false);","handlingStrategy":"retry","validationCode":"// before resubmitting with an explicit id, confirm it is not in use\nJobMetrics m = jobClient.getJobMetrics(jobId);\nif (m != null && !m.equals(JobMetrics.empty())) {\n    throw new IllegalStateException(\"jobId in use; generate a new one or wait for cleanup\");\n}","typeGuard":null,"tryCatchPattern":"for (int i = 0; i < 3; i++) {\n    try { client.submitJob(jobId, config, false); break; }\n    catch (JobException e) {\n        if (e.getMessage().contains(\"waiting for terminal state cleanup\")) {\n            Thread.sleep(5000L * (i + 1)); continue;\n        }\n        throw e;\n    }\n}","preventionTips":["Prefer engine-generated job ids over hardcoded ones.","After stopping a job, poll until terminal state cleanup completes before resubmitting.","Always submit restores with start-with-savepoint enabled.","Add backoff to automated resubmission loops."],"tags":["zeta-engine","job-submission","jobid-reuse"],"backgroundTag":"invalid-state-transition","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}