What causes "Recover workflow instance failed: %s" in apache/dolphinscheduler?

Inspect recoverFailureTaskResponse.getMessage() in the exception and the master logs for root cause

apache/dolphinscheduler · error · ServiceException

Recover workflow instance failed: %s

Error message

Recover workflow instance failed: %s

What it means

Thrown when the master's response to triggerFromFailureTasks reports failure. The recovery request was delivered to a master, but the master could not restart the failed tasks; its error message is embedded in this exception.

Source

Thrown at dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/RecoverFailureTaskInstanceExecutorDelegate.java:68

                            workflowInstance.getName(), workflowInstance.getState()));
        }

        final Server masterServer = registryClient.getRandomServer(RegistryNodeType.MASTER).orElse(null);
        if (masterServer == null) {
            throw new ServiceException("no master server available");
        }
        final WorkflowInstanceRecoverFailureTasksRequest recoverFailureTaskRequest =
                WorkflowInstanceRecoverFailureTasksRequest.builder()
                        .workflowInstanceId(workflowInstance.getId())
                        .userId(recoverFailureTaskInstanceOperation.executeUser.getId())
                        .build();

        final WorkflowInstanceRecoverFailureTasksResponse recoverFailureTaskResponse = Clients
                .withService(IWorkflowControlClient.class)
                .withHost(masterServer.getHost() + ":" + masterServer.getPort())
                .triggerFromFailureTasks(recoverFailureTaskRequest);
        if (!recoverFailureTaskResponse.isSuccess()) {
            throw new ServiceException("Recover workflow instance failed: " + recoverFailureTaskResponse.getMessage());
        }
        return null;
    }

    @Getter
    public static class RecoverFailureTaskInstanceOperation {

        private final RecoverFailureTaskInstanceExecutorDelegate recoverFailureTaskInstanceExecutorDelegate;

        private WorkflowInstance workflowInstance;

        private User executeUser;

        public RecoverFailureTaskInstanceOperation(RecoverFailureTaskInstanceExecutorDelegate recoverFailureTaskInstanceExecutorDelegate) {
            this.recoverFailureTaskInstanceExecutorDelegate = recoverFailureTaskInstanceExecutorDelegate;
        }

        public RecoverFailureTaskInstanceOperation onWorkflowInstance(WorkflowInstance workflowInstance) {

View on GitHub (pinned to 02eac45a1b)

Solutions

Inspect recoverFailureTaskResponse.getMessage() in the exception and the master logs for root cause
Ensure no other control operation is running on the instance; retry after it completes
Verify the workflow definition is still valid and compatible
Check master DB health and command queue; retry recovery once resolved

Example fix

// before: immediate recover after failure detected
api.recoverFailureTasks(instanceId, userId);
// after: ensure instance is idle/not being operated on, then recover
waitUntilNoPendingOperations(instanceId);
api.recoverFailureTasks(instanceId, userId);

Defensive patterns

Strategy: retry

Validate before calling

// ensure no concurrent control operation on the instance
boolean idle = !operationLockHeld(workflowInstance.getId());

Try / catch

try { delegate.recoverFailureTasks(op); } catch (ServiceException e) { log.error("Recover rejected by master: {}", e.getMessage()); retryWithBackoff(op); }

Prevention

Serialize control operations per workflow instance
Inspect master response message and logs for the root cause
Keep definitions valid and compatible across versions
Check master DB/command queue health before bulk recoveries

When it happens

Trigger: RecoverFailureTaskInstanceExecutorDelegate.execute invokes triggerFromFailureTasks on the master host and recoverFailureTaskResponse.isSuccess() is false — master-side rejection such as instance command insert failure, DB errors, or concurrency conflicts (instance already being operated on).

Common situations: Another operation (pause/kill/recovery) is in flight for the same instance; workflow definition changed since failure so recovery is rejected; master DB or command-queue issues; API/master version mismatch.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/7023d391460f8012. Report an issue: GitHub.