apache/seatunnel · error · JobException

The job id is waiting for terminal state cleanup, please…

Error message

The job id %s is waiting for terminal state cleanup, please retry later.

What it means

When a job is submitted with an explicit jobId, the engine checks whether a pending terminal-state cleanup record still owned by that jobId exists in the job state IMap. If one exists and the submission is not a savepoint (restore) start, the submission is rejected with this JobException, telling the caller to retry later. It is a transient rejection while the previous job's end-of-life cleanup finishes.

Solutions

Wait and retry the submission until the previous job's cleanup completes.
Let the engine generate the jobId (omit the explicit id) so a fresh id is used.
If this is a restore, submit with start-with-savepoint so cleanupPendingJobStateForRestore runs instead of rejecting.
Poll job status/history via CLI or REST to confirm the old job fully terminated before resubmitting.

Example fix

// before: reused fixed id
long jobId = 861912345L;
client.submitJob(jobId, config, false);
// after: retry with backoff on a fresh id
long jobId = new Random().nextLong();
client.submitJob(jobId, config, false);

Defensive patterns

Strategy: retry

Validate before calling

// before resubmitting with an explicit id, confirm it is not in use
JobMetrics m = jobClient.getJobMetrics(jobId);
if (m != null && !m.equals(JobMetrics.empty())) {
    throw new IllegalStateException("jobId in use; generate a new one or wait for cleanup");
}

Try / catch

for (int i = 0; i < 3; i++) {
    try { client.submitJob(jobId, config, false); break; }
    catch (JobException e) {
        if (e.getMessage().contains("waiting for terminal state cleanup")) {
            Thread.sleep(5000L * (i + 1)); continue;
        }
        throw e;
    }
}

Prevention

Prefer engine-generated job ids over hardcoded ones.
After stopping a job, poll until terminal state cleanup completes before resubmitting.
Always submit restores with start-with-savepoint enabled.
Add backoff to automated resubmission loops.

When it happens

Trigger: submitJob with a reused jobId while a pendingCleanupRecord for that id still exists (previous job awaiting cleanup) and isStartWithSavePoint is false.

Common situations: Hardcoded jobId resubmitted immediately after stopping the previous job; automated resubmission loops that don't wait for cleanup; fast restart after stop/savepoint with the same fixed id.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/fabcdb76974624c2. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java:1419

                    JobInfo submittedJobInfo = null;
                    try {
                        JobImmutableInformation submittedJobImmutableInformation =
                                deserializeJobImmutableInformation(jobImmutableInformation);
                        validateCheckpointRestoreSourceJobIsTerminal(
                                submittedJobImmutableInformation, jobId);
                        if (isStartWithSavePoint) {
                            cleanupPendingPipelineCleanupForRestore(jobId);
                        }
                        JobCleanupRecord pendingCleanupRecord =
                                pendingJobCleanupIMap != null
                                        ? pendingJobCleanupIMap.get(jobId)
                                        : null;
                        if (pendingCleanupRecord != null
                                && isCleanupOwnedByCurrentJob(jobId, pendingCleanupRecord)) {
                            if (isStartWithSavePoint) {
                                cleanupPendingJobStateForRestore(jobId, pendingCleanupRecord);
                            } else {
                                throw new JobException(
                                        String.format(
                                                "The job id %s is waiting for terminal state cleanup, please retry later.",
                                                jobId));
                            }
                        }

                        jobMaster =
                                new JobMaster(
                                        jobId,
                                        jobImmutableInformation,
                                        this.nodeEngine,
                                        mdcExecutorService,
                                        getResourceManager(),
                                        getJobHistoryService(),
                                        runningJobStateIMap,
                                        runningJobStateTimestampsIMap,
                                        ownedSlotProfilesIMap,
                                        runningJobInfoIMap,

View on GitHub (pinned to cf67b549a7)