{"record":{"id":"a0237e6eda745d5c","repo":"apache/dolphinscheduler","slug":"no-master-server-available-a0237e","errorCode":null,"errorMessage":"no master server available","messagePattern":"no master server available","errorType":"exception","errorClass":"ServiceException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/RecoverFailureTaskInstanceExecutorDelegate.java","lineNumber":55,"sourceCode":"public class RecoverFailureTaskInstanceExecutorDelegate\n        implements\n            IExecutorDelegate<RecoverFailureTaskInstanceExecutorDelegate.RecoverFailureTaskInstanceOperation, Void> {\n\n    @Autowired\n    private RegistryClient registryClient;\n\n    @Override\n    public Void execute(RecoverFailureTaskInstanceOperation recoverFailureTaskInstanceOperation) {\n        WorkflowInstance workflowInstance = recoverFailureTaskInstanceOperation.getWorkflowInstance();\n        if (!workflowInstance.getState().isFailure()) {\n            throw new ServiceException(\n                    String.format(\"The workflow instance: %s status is %s, can not be recovered\",\n                            workflowInstance.getName(), workflowInstance.getState()));\n        }\n\n        final Server masterServer = registryClient.getRandomServer(RegistryNodeType.MASTER).orElse(null);\n        if (masterServer == null) {\n            throw new ServiceException(\"no master server available\");\n        }\n        final WorkflowInstanceRecoverFailureTasksRequest recoverFailureTaskRequest =\n                WorkflowInstanceRecoverFailureTasksRequest.builder()\n                        .workflowInstanceId(workflowInstance.getId())\n                        .userId(recoverFailureTaskInstanceOperation.executeUser.getId())\n                        .build();\n\n        final WorkflowInstanceRecoverFailureTasksResponse recoverFailureTaskResponse = Clients\n                .withService(IWorkflowControlClient.class)\n                .withHost(masterServer.getHost() + \":\" + masterServer.getPort())\n                .triggerFromFailureTasks(recoverFailureTaskRequest);\n        if (!recoverFailureTaskResponse.isSuccess()) {\n            throw new ServiceException(\"Recover workflow instance failed: \" + recoverFailureTaskResponse.getMessage());\n        }\n        return null;\n    }\n\n    @Getter","sourceCodeStart":37,"sourceCodeEnd":73,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/RecoverFailureTaskInstanceExecutorDelegate.java#L37-L73","documentation":"Thrown during failed-task recovery when the registry contains no master server. The API must forward the WorkflowInstanceRecoverFailureTasksRequest to a master; with zero registered masters it cannot proceed and throws immediately.","triggerScenarios":"RecoverFailureTaskInstanceExecutorDelegate.execute reaches registryClient.getRandomServer(RegistryNodeType.MASTER) and the optional is empty (no live masters registered).","commonSituations":"Entire master tier down after maintenance; masters unable to connect to ZooKeeper; registry quorum outage so the API sees an empty server list; scaled-down cluster with only API/worker nodes.","solutions":["Start or restart at least one MasterServer and confirm registration in ZooKeeper","Verify masters' and API's registry (ZooKeeper) endpoints are consistent","Check ZooKeeper health and network reachability from both API and masters","Retry recovery after the master tier is healthy"],"exampleFix":"// before\napi.recoverFailureTasks(instanceId, userId); // 500 if masters down\n// after\nwaitForMasterAvailable(registryClient); // poll getRandomServer(MASTER)\napi.recoverFailureTasks(instanceId, userId);","handlingStrategy":"validation","validationCode":"boolean masterAvailable = registryClient.getRandomServer(RegistryNodeType.MASTER).isPresent();\nif (!masterAvailable) throw new IllegalStateException(\"No master registered; cannot recover failed tasks\");","typeGuard":"boolean hasMaster(RegistryClient c) { return c.getRandomServer(RegistryNodeType.MASTER).isPresent(); }","tryCatchPattern":"try { delegate.recoverFailureTasks(op); } catch (ServiceException e) { if (e.getMessage().contains(\"no master server available\")) { waitForMasterAndRetry(op); } else { throw e; } }","preventionTips":["Alert on master count in ZooKeeper dropping to zero","Gate recovery automations on cluster health checks","Ensure masters auto-register and heartbeat correctly after restarts","Verify ZooKeeper quorum health regularly"],"tags":["dolphinscheduler","master-unavailable","registry","cluster-health"],"backgroundTag":"connection-refused","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}