{"record":{"id":"04ceaa0bbe5bf455","repo":"apache/seatunnel","slug":"job-s-not-running-restore-in-progress","errorCode":null,"errorMessage":"Job %s not running (restore in progress)","messagePattern":"Job (.+?) not running \\(restore in progress\\)","errorType":"exception","errorClass":"JobRestoreInProgressException","httpStatus":null,"severity":"warning","filePath":"seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java","lineNumber":2074,"sourceCode":"     * TaskGroup's state.\n     */\n    public void updateTaskExecutionState(TaskExecutionState taskExecutionState) {\n        logger.info(\n                String.format(\n                        \"Received task end from execution %s, state %s\",\n                        taskExecutionState.getTaskGroupLocation(),\n                        taskExecutionState.getExecutionState()));\n        TaskGroupLocation taskGroupLocation = taskExecutionState.getTaskGroupLocation();\n        JobMaster runningJobMaster = runningJobMasterMap.get(taskGroupLocation.getJobId());\n\n        if (runningJobMaster == null && !restoreAllJobFromMasterNodeSwitchFuture.isDone()) {\n            // Restore still in progress, return early and let worker retry\n            // This is acceptable because worker already has retry logic\n            logger.info(\n                    String.format(\n                            \"Job %s not found and restore still in progress, worker will retry\",\n                            taskGroupLocation.getJobId()));\n            throw new JobRestoreInProgressException(\n                    String.format(\n                            \"Job %s not running (restore in progress)\",\n                            taskGroupLocation.getJobId()));\n        }\n        if (runningJobMaster == null) {\n            throw new JobNotFoundException(\n                    String.format(\"Job %s not running\", taskGroupLocation.getJobId()));\n        }\n        runningJobMaster.updateTaskExecutionState(taskExecutionState);\n    }\n\n    public void shutdown() {\n        isActive = false;\n        if (masterActiveListener != null) {\n            masterActiveListener.shutdown();\n        }\n        if (metricsFetchExecutor != null) {\n            metricsFetchExecutor.shutdownNow();","sourceCodeStart":2056,"sourceCodeEnd":2092,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java#L2056-L2092","documentation":"updateTaskExecutionState throws JobRestoreInProgressException when a task reports its state but the JobMaster for that job is not yet running because the job's state restore (after failover/restart) is still in progress. The exception is deliberately thrown early so the worker node retries the update later, since worker-side retry logic exists.","triggerScenarios":"A TaskExecutionService on a worker calls the coordinator's UpdateTaskExecutionStateOperation for a task group whose jobId is registered as restoring but has no active runningJobMaster yet — typically during cluster failover, master node switch, or job restore from checkpoint while tasks are still executing.","commonSituations":"Master node crash followed by new master restoring jobs from checkpoint storage; worker tasks report state transitions during the restore window; large DAGs with slow restore making the window noticeable.","solutions":["No action needed if seen transiently: the worker retries the update automatically after the restore completes.","If it persists, check master logs for restore progress/errors (restoreJobDAGInfo, checkpoint recovery) and ensure checkpoint storage is reachable.","Reduce restore time by lowering concurrent job restore load or verifying the checkpoint storage (HDFS/S3) latency and availability."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (JobRestoreInProgressException e) {\n    // transient: back off and retry until the JobMaster is restored\n    Uninterruptibles.sleepUninterruptibly(200, TimeUnit.MILLISECONDS);\n    retryUpdateTaskState(taskExecutionState);\n}","preventionTips":["Expect this during master failover; build worker-side idempotent retry around task state updates","Keep checkpoint storage fast and available to shorten the restore window","Monitor restore duration metrics to distinguish transient vs stuck restores"],"tags":["seatunnel","zeta-engine","failover","restore","retryable"],"backgroundTag":"invalid-state-transition","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}