apache/dolphinscheduler · error · ServiceException

Recover workflow instance failed: %s

Error message

Recover workflow instance failed: %s

What it means

Thrown by RecoverSuspendedWorkflowInstanceExecutorDelegate.execute when the master server's IWorkflowControlClient responds to a triggerFromSuspendTasks request with a non-success response. The API module asked the master to resume a suspended workflow instance from its suspended tasks, and the master reported failure; the master's message is appended. It indicates the resume operation was rejected or failed server-side.

Source

Thrown at dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/RecoverSuspendedWorkflowInstanceExecutorDelegate.java:65

                    String.format("The workflow instance: %s state is %s, cannot recovery", workflowInstance.getName(),
                            workflowInstance.getState()));
        }
        final Server masterServer = registryClient.getRandomServer(RegistryNodeType.MASTER).orElse(null);
        if (masterServer == null) {
            throw new ServiceException("no master server available");
        }
        final WorkflowInstanceRecoverSuspendTasksRequest recoverSuspendTaskRequest =
                WorkflowInstanceRecoverSuspendTasksRequest.builder()
                        .workflowInstanceId(workflowInstance.getId())
                        .userId(workflowInstanceControlRequest.executeUser.getId())
                        .build();

        final WorkflowInstanceRecoverSuspendTasksResponse recoverSuspendTaskResponse = Clients
                .withService(IWorkflowControlClient.class)
                .withHost(masterServer.getHost() + ":" + masterServer.getPort())
                .triggerFromSuspendTasks(recoverSuspendTaskRequest);
        if (!recoverSuspendTaskResponse.isSuccess()) {
            throw new ServiceException("Recover workflow instance failed: " + recoverSuspendTaskResponse.getMessage());
        }
        return null;
    }

    public static class RecoverSuspendedWorkflowInstanceOperation {

        private final RecoverSuspendedWorkflowInstanceExecutorDelegate recoverSuspendedWorkflowInstanceExecutorDelegate;

        private WorkflowInstance workflowInstance;

        private User executeUser;

        public RecoverSuspendedWorkflowInstanceOperation(RecoverSuspendedWorkflowInstanceExecutorDelegate recoverSuspendedWorkflowInstanceExecutorDelegate) {
            this.recoverSuspendedWorkflowInstanceExecutorDelegate = recoverSuspendedWorkflowInstanceExecutorDelegate;
        }

        public RecoverSuspendedWorkflowInstanceExecutorDelegate.RecoverSuspendedWorkflowInstanceOperation onWorkflowInstance(WorkflowInstance workflowInstance) {
            this.workflowInstance = workflowInstance;

View on GitHub (pinned to 02eac45a1b)

Solutions

  1. Read the appended recoverSuspendTaskResponse.getMessage() for the master-side cause and fix that underlying issue
  2. Refresh the workflow instance list and verify the instance is still in a suspended state before recovering
  3. Retry the recover operation; transient master issues can cause failures
  4. Check master server logs around the request time for the root failure
  5. Verify the master server the API contacted is healthy and holds the instance

Example fix

// before
Clients.withService(IWorkflowControlClient.class)
    .withHost(masterServer.getHost() + ":" + masterServer.getPort())
    .triggerFromSuspendTasks(recoverSuspendTaskRequest);
// after
// check state client-side before triggering
if (!workflowInstance.getState().isSuspend()) {
    throw new ServiceException("Instance is not suspended, cannot recover");
}
Defensive patterns

Strategy: try-catch

Validate before calling

WorkflowInstance wi = getFreshInstance(id);
if (wi == null || wi.getState() == null || !wi.getState().isSuspend()) {
    throw new IllegalArgumentException("Instance not in suspended state");
}

Try / catch

try {
    recoverExecutorDelegate.execute(op);
} catch (ServiceException e) {
    log.error("Recover failed: {}", e.getMessage(), e);
    // refresh instance state and surface master-side message to user
}

Prevention

When it happens

Trigger: Calling recover on a suspended workflow instance where the master's triggerFromSuspendTasks returns success=false, e.g. the workflow instance was deleted or modified between request and execution, the suspended task no longer exists, or the master rejects the recover due to invalid instance state.

Common situations: Recovering a workflow instance that another user already stopped or recovered; racing with master failover so the instance's host changed; stale UI data pointing at an old suspended instance; master-side command generation errors.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/cbe618e9ad6d920d. Report an issue: GitHub.