What causes "The workflow instance: %s status is %s, can not be recovered" in apache/dolphinscheduler?

Query the instance's current state first; only recover when state is failure

apache/dolphinscheduler · warning · ServiceException

The workflow instance: %s status is %s, can not be recovered

Error message

The workflow instance: %s status is %s, can not be recovered

What it means

Thrown when attempting to recover (rerun) failed tasks of a workflow instance that is not in a failure state. Recovery from failed tasks is only valid when workflowInstance.getState().isFailure(); otherwise the API refuses the operation.

Source

Thrown at dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/RecoverFailureTaskInstanceExecutorDelegate.java:48

import lombok.Getter;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

@Component
public class RecoverFailureTaskInstanceExecutorDelegate
        implements
            IExecutorDelegate<RecoverFailureTaskInstanceExecutorDelegate.RecoverFailureTaskInstanceOperation, Void> {

    @Autowired
    private RegistryClient registryClient;

    @Override
    public Void execute(RecoverFailureTaskInstanceOperation recoverFailureTaskInstanceOperation) {
        WorkflowInstance workflowInstance = recoverFailureTaskInstanceOperation.getWorkflowInstance();
        if (!workflowInstance.getState().isFailure()) {
            throw new ServiceException(
                    String.format("The workflow instance: %s status is %s, can not be recovered",
                            workflowInstance.getName(), workflowInstance.getState()));
        }

        final Server masterServer = registryClient.getRandomServer(RegistryNodeType.MASTER).orElse(null);
        if (masterServer == null) {
            throw new ServiceException("no master server available");
        }
        final WorkflowInstanceRecoverFailureTasksRequest recoverFailureTaskRequest =
                WorkflowInstanceRecoverFailureTasksRequest.builder()
                        .workflowInstanceId(workflowInstance.getId())
                        .userId(recoverFailureTaskInstanceOperation.executeUser.getId())
                        .build();

        final WorkflowInstanceRecoverFailureTasksResponse recoverFailureTaskResponse = Clients
                .withService(IWorkflowControlClient.class)
                .withHost(masterServer.getHost() + ":" + masterServer.getPort())
                .triggerFromFailureTasks(recoverFailureTaskRequest);

View on GitHub (pinned to 02eac45a1b)

Solutions

Query the instance's current state first; only recover when state is failure
Refresh the UI/instance data — the state may have changed since it was displayed
If the instance is killed/stopped, use the appropriate resume/restart operation instead
Fix the underlying failure and use the correct operation for the current state

Example fix

// before
api.recoverFailureTasks(instanceId, userId);
// after
WorkflowInstance wf = api.queryWorkflowInstance(instanceId);
if (wf.getState().isFailure()) {
    api.recoverFailureTasks(instanceId, userId);
}

Defensive patterns

Strategy: validation

Validate before calling

if (!workflowInstance.getState().isFailure()) {
    throw new IllegalStateException("Instance state " + workflowInstance.getState() + " is not recoverable from failure");
}

Type guard

boolean recoverableFromFailure(WorkflowInstance wf) { return wf.getState().isFailure(); }

Try / catch

try { delegate.recoverFailureTasks(op); } catch (ServiceException e) { if (e.getMessage().contains("can not be recovered")) { log.info("Instance not in failure state; skipping recovery"); } else { throw e; } }

Prevention

Check state immediately before recovery calls
Refresh instance data after any state-changing operation
Route failed instances to recover-failure and killed instances to their own recovery path
Avoid caching instance state in long-running automation loops

When it happens

Trigger: Calling the recover-failed-tasks API (RecoverFailureTaskInstanceExecutorDelegate.execute) on an instance whose state is running, success, stopped, killed, paused, etc.

Common situations: User clicks 'recover failed task' after the workflow already succeeded or was killed; UI state stale relative to actual instance state; script retries recovery on an already-recovered instance; confusing 'workflow success despite task failure' cases where the state is not FAILURE.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/7fe6e72b5dfc2bf5. Report an issue: GitHub.