apache/dolphinscheduler · error · ServiceException

no master server available

Error message

no master server available

What it means

Thrown during failed-task recovery when the registry contains no master server. The API must forward the WorkflowInstanceRecoverFailureTasksRequest to a master; with zero registered masters it cannot proceed and throws immediately.

Source

Thrown at dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/RecoverFailureTaskInstanceExecutorDelegate.java:55

public class RecoverFailureTaskInstanceExecutorDelegate
        implements
            IExecutorDelegate<RecoverFailureTaskInstanceExecutorDelegate.RecoverFailureTaskInstanceOperation, Void> {

    @Autowired
    private RegistryClient registryClient;

    @Override
    public Void execute(RecoverFailureTaskInstanceOperation recoverFailureTaskInstanceOperation) {
        WorkflowInstance workflowInstance = recoverFailureTaskInstanceOperation.getWorkflowInstance();
        if (!workflowInstance.getState().isFailure()) {
            throw new ServiceException(
                    String.format("The workflow instance: %s status is %s, can not be recovered",
                            workflowInstance.getName(), workflowInstance.getState()));
        }

        final Server masterServer = registryClient.getRandomServer(RegistryNodeType.MASTER).orElse(null);
        if (masterServer == null) {
            throw new ServiceException("no master server available");
        }
        final WorkflowInstanceRecoverFailureTasksRequest recoverFailureTaskRequest =
                WorkflowInstanceRecoverFailureTasksRequest.builder()
                        .workflowInstanceId(workflowInstance.getId())
                        .userId(recoverFailureTaskInstanceOperation.executeUser.getId())
                        .build();

        final WorkflowInstanceRecoverFailureTasksResponse recoverFailureTaskResponse = Clients
                .withService(IWorkflowControlClient.class)
                .withHost(masterServer.getHost() + ":" + masterServer.getPort())
                .triggerFromFailureTasks(recoverFailureTaskRequest);
        if (!recoverFailureTaskResponse.isSuccess()) {
            throw new ServiceException("Recover workflow instance failed: " + recoverFailureTaskResponse.getMessage());
        }
        return null;
    }

    @Getter

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Start or restart at least one MasterServer and confirm registration in ZooKeeper
2. Verify masters' and API's registry (ZooKeeper) endpoints are consistent
3. Check ZooKeeper health and network reachability from both API and masters
4. Retry recovery after the master tier is healthy

Example fix

// before
api.recoverFailureTasks(instanceId, userId); // 500 if masters down
// after
waitForMasterAvailable(registryClient); // poll getRandomServer(MASTER)
api.recoverFailureTasks(instanceId, userId);

boolean masterAvailable = registryClient.getRandomServer(RegistryNodeType.MASTER).isPresent();
if (!masterAvailable) throw new IllegalStateException("No master registered; cannot recover failed tasks");

boolean hasMaster(RegistryClient c) { return c.getRandomServer(RegistryNodeType.MASTER).isPresent(); }

try { delegate.recoverFailureTasks(op); } catch (ServiceException e) { if (e.getMessage().contains("no master server available")) { waitForMasterAndRetry(op); } else { throw e; } }

When it happens

Trigger: RecoverFailureTaskInstanceExecutorDelegate.execute reaches registryClient.getRandomServer(RegistryNodeType.MASTER) and the optional is empty (no live masters registered).

Common situations: Entire master tier down after maintenance; masters unable to connect to ZooKeeper; registry quorum outage so the API sees an empty server list; scaled-down cluster with only API/worker nodes.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• no master server available
• no master server available
• Backfill workflow failed: %s
• The workflow instance: %s status is %s, can not pause
• WorkflowInstance: %s pause failed: %s

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/a0237e6eda745d5c. Report an issue: GitHub.