apache/seatunnel · error · JobException

The job id %s is waiting for terminal state cleanup, please

Error message

The job id %s is waiting for terminal state cleanup, please retry later.

What it means

When a job is submitted with an explicit jobId, the engine checks whether a pending terminal-state cleanup record still owned by that jobId exists in the job state IMap. If one exists and the submission is not a savepoint (restore) start, the submission is rejected with this JobException, telling the caller to retry later. It is a transient rejection while the previous job's end-of-life cleanup finishes.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java:1419

                    JobInfo submittedJobInfo = null;
                    try {
                        JobImmutableInformation submittedJobImmutableInformation =
                                deserializeJobImmutableInformation(jobImmutableInformation);
                        validateCheckpointRestoreSourceJobIsTerminal(
                                submittedJobImmutableInformation, jobId);
                        if (isStartWithSavePoint) {
                            cleanupPendingPipelineCleanupForRestore(jobId);
                        }
                        JobCleanupRecord pendingCleanupRecord =
                                pendingJobCleanupIMap != null
                                        ? pendingJobCleanupIMap.get(jobId)
                                        : null;
                        if (pendingCleanupRecord != null
                                && isCleanupOwnedByCurrentJob(jobId, pendingCleanupRecord)) {
                            if (isStartWithSavePoint) {
                                cleanupPendingJobStateForRestore(jobId, pendingCleanupRecord);
                            } else {
                                throw new JobException(
                                        String.format(
                                                "The job id %s is waiting for terminal state cleanup, please retry later.",
                                                jobId));
                            }
                        }

                        jobMaster =
                                new JobMaster(
                                        jobId,
                                        jobImmutableInformation,
                                        this.nodeEngine,
                                        mdcExecutorService,
                                        getResourceManager(),
                                        getJobHistoryService(),
                                        runningJobStateIMap,
                                        runningJobStateTimestampsIMap,
                                        ownedSlotProfilesIMap,
                                        runningJobInfoIMap,

View on GitHub (pinned to cf67b549a7)

Solutions

Wait and retry the submission until the previous job's cleanup completes.
Let the engine generate the jobId (omit the explicit id) so a fresh id is used.
If this is a restore, submit with start-with-savepoint so cleanupPendingJobStateForRestore runs instead of rejecting.
Poll job status/history via CLI or REST to confirm the old job fully terminated before resubmitting.

Example fix

// before: reused fixed id
long jobId = 861912345L;
client.submitJob(jobId, config, false);
// after: retry with backoff on a fresh id
long jobId = new Random().nextLong();
client.submitJob(jobId, config, false);

Defensive patterns

Strategy: retry

Validate before calling

// before resubmitting with an explicit id, confirm it is not in use
JobMetrics m = jobClient.getJobMetrics(jobId);
if (m != null && !m.equals(JobMetrics.empty())) {
    throw new IllegalStateException("jobId in use; generate a new one or wait for cleanup");
}

Try / catch

for (int i = 0; i < 3; i++) {
    try { client.submitJob(jobId, config, false); break; }
    catch (JobException e) {
        if (e.getMessage().contains("waiting for terminal state cleanup")) {
            Thread.sleep(5000L * (i + 1)); continue;
        }
        throw e;
    }
}

Prevention

Prefer engine-generated job ids over hardcoded ones.
After stopping a job, poll until terminal state cleanup completes before resubmitting.
Always submit restores with start-with-savepoint enabled.
Add backoff to automated resubmission loops.

When it happens

Trigger: submitJob with a reused jobId while a pendingCleanupRecord for that id still exists (previous job awaiting cleanup) and isStartWithSavePoint is false.

Common situations: Hardcoded jobId resubmitted immediately after stopping the previous job; automated resubmission loops that don't wait for cleanup; fast restart after stop/savepoint with the same fixed id.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/fabcdb76974624c2. Report an issue: GitHub.