{"record":{"id":"7023d391460f8012","repo":"apache/dolphinscheduler","slug":"recover-workflow-instance-failed-s","errorCode":null,"errorMessage":"Recover workflow instance failed: %s","messagePattern":"Recover workflow instance failed: (.+?)","errorType":"exception","errorClass":"ServiceException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/RecoverFailureTaskInstanceExecutorDelegate.java","lineNumber":68,"sourceCode":"                            workflowInstance.getName(), workflowInstance.getState()));\n        }\n\n        final Server masterServer = registryClient.getRandomServer(RegistryNodeType.MASTER).orElse(null);\n        if (masterServer == null) {\n            throw new ServiceException(\"no master server available\");\n        }\n        final WorkflowInstanceRecoverFailureTasksRequest recoverFailureTaskRequest =\n                WorkflowInstanceRecoverFailureTasksRequest.builder()\n                        .workflowInstanceId(workflowInstance.getId())\n                        .userId(recoverFailureTaskInstanceOperation.executeUser.getId())\n                        .build();\n\n        final WorkflowInstanceRecoverFailureTasksResponse recoverFailureTaskResponse = Clients\n                .withService(IWorkflowControlClient.class)\n                .withHost(masterServer.getHost() + \":\" + masterServer.getPort())\n                .triggerFromFailureTasks(recoverFailureTaskRequest);\n        if (!recoverFailureTaskResponse.isSuccess()) {\n            throw new ServiceException(\"Recover workflow instance failed: \" + recoverFailureTaskResponse.getMessage());\n        }\n        return null;\n    }\n\n    @Getter\n    public static class RecoverFailureTaskInstanceOperation {\n\n        private final RecoverFailureTaskInstanceExecutorDelegate recoverFailureTaskInstanceExecutorDelegate;\n\n        private WorkflowInstance workflowInstance;\n\n        private User executeUser;\n\n        public RecoverFailureTaskInstanceOperation(RecoverFailureTaskInstanceExecutorDelegate recoverFailureTaskInstanceExecutorDelegate) {\n            this.recoverFailureTaskInstanceExecutorDelegate = recoverFailureTaskInstanceExecutorDelegate;\n        }\n\n        public RecoverFailureTaskInstanceOperation onWorkflowInstance(WorkflowInstance workflowInstance) {","sourceCodeStart":50,"sourceCodeEnd":86,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/RecoverFailureTaskInstanceExecutorDelegate.java#L50-L86","documentation":"Thrown when the master's response to triggerFromFailureTasks reports failure. The recovery request was delivered to a master, but the master could not restart the failed tasks; its error message is embedded in this exception.","triggerScenarios":"RecoverFailureTaskInstanceExecutorDelegate.execute invokes triggerFromFailureTasks on the master host and recoverFailureTaskResponse.isSuccess() is false — master-side rejection such as instance command insert failure, DB errors, or concurrency conflicts (instance already being operated on).","commonSituations":"Another operation (pause/kill/recovery) is in flight for the same instance; workflow definition changed since failure so recovery is rejected; master DB or command-queue issues; API/master version mismatch.","solutions":["Inspect recoverFailureTaskResponse.getMessage() in the exception and the master logs for root cause","Ensure no other control operation is running on the instance; retry after it completes","Verify the workflow definition is still valid and compatible","Check master DB health and command queue; retry recovery once resolved"],"exampleFix":"// before: immediate recover after failure detected\napi.recoverFailureTasks(instanceId, userId);\n// after: ensure instance is idle/not being operated on, then recover\nwaitUntilNoPendingOperations(instanceId);\napi.recoverFailureTasks(instanceId, userId);","handlingStrategy":"retry","validationCode":"// ensure no concurrent control operation on the instance\nboolean idle = !operationLockHeld(workflowInstance.getId());","typeGuard":null,"tryCatchPattern":"try { delegate.recoverFailureTasks(op); } catch (ServiceException e) { log.error(\"Recover rejected by master: {}\", e.getMessage()); retryWithBackoff(op); }","preventionTips":["Serialize control operations per workflow instance","Inspect master response message and logs for the root cause","Keep definitions valid and compatible across versions","Check master DB/command queue health before bulk recoveries"],"tags":["dolphinscheduler","master-rpc","recovery","api-error"],"backgroundTag":"api-error-response","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}