apache/seatunnel · warning · JobRestoreInProgressException

Job %s not running (restore in progress)

Error message

Job %s not running (restore in progress)

What it means

updateTaskExecutionState throws JobRestoreInProgressException when a task reports its state but the JobMaster for that job is not yet running because the job's state restore (after failover/restart) is still in progress. The exception is deliberately thrown early so the worker node retries the update later, since worker-side retry logic exists.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java:2074

     * TaskGroup's state.
     */
    public void updateTaskExecutionState(TaskExecutionState taskExecutionState) {
        logger.info(
                String.format(
                        "Received task end from execution %s, state %s",
                        taskExecutionState.getTaskGroupLocation(),
                        taskExecutionState.getExecutionState()));
        TaskGroupLocation taskGroupLocation = taskExecutionState.getTaskGroupLocation();
        JobMaster runningJobMaster = runningJobMasterMap.get(taskGroupLocation.getJobId());

        if (runningJobMaster == null && !restoreAllJobFromMasterNodeSwitchFuture.isDone()) {
            // Restore still in progress, return early and let worker retry
            // This is acceptable because worker already has retry logic
            logger.info(
                    String.format(
                            "Job %s not found and restore still in progress, worker will retry",
                            taskGroupLocation.getJobId()));
            throw new JobRestoreInProgressException(
                    String.format(
                            "Job %s not running (restore in progress)",
                            taskGroupLocation.getJobId()));
        }
        if (runningJobMaster == null) {
            throw new JobNotFoundException(
                    String.format("Job %s not running", taskGroupLocation.getJobId()));
        }
        runningJobMaster.updateTaskExecutionState(taskExecutionState);
    }

    public void shutdown() {
        isActive = false;
        if (masterActiveListener != null) {
            masterActiveListener.shutdown();
        }
        if (metricsFetchExecutor != null) {
            metricsFetchExecutor.shutdownNow();

View on GitHub (pinned to cf67b549a7)

Solutions

1. No action needed if seen transiently: the worker retries the update automatically after the restore completes.
2. If it persists, check master logs for restore progress/errors (restoreJobDAGInfo, checkpoint recovery) and ensure checkpoint storage is reachable.
3. Reduce restore time by lowering concurrent job restore load or verifying the checkpoint storage (HDFS/S3) latency and availability.

catch (JobRestoreInProgressException e) {
    // transient: back off and retry until the JobMaster is restored
    Uninterruptibles.sleepUninterruptibly(200, TimeUnit.MILLISECONDS);
    retryUpdateTaskState(taskExecutionState);
}

When it happens

Trigger: A TaskExecutionService on a worker calls the coordinator's UpdateTaskExecutionStateOperation for a task group whose jobId is registered as restoring but has no active runningJobMaster yet — typically during cluster failover, master node switch, or job restore from checkpoint while tasks are still executing.

Common situations: Master node crash followed by new master restoring jobs from checkpoint storage; worker tasks report state transitions during the restore window; large DAGs with slow restore making the window noticeable.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Can not get coordinator service from an active master node.
• Failed to fetch running jobs from IMap during master switch
• wait worker register error
• Job id %s restore failed, can not get job state
• Job id %s init failed

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/04ceaa0bbe5bf455. Report an issue: GitHub.