apache/dolphinscheduler · error · ServiceException

WorkflowInstance: %s stop failed: %s

Error message

WorkflowInstance: %s stop failed: %s

What it means

Thrown by StopWorkflowInstanceExecutorDelegate.stopInMaster when the master's stopWorkflowInstance response is null or reports success=false. The stop request reached the instance's host master but the stop operation failed; the whole response object is appended to the message.

Source

Thrown at dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/StopWorkflowInstanceExecutorDelegate.java:101

            return null;
        });
        log.info("Update workflow instance {} state from: {} to {} success",
                workflowInstance.getName(),
                workflowInstance.getState().name(),
                WorkflowExecutionStatus.STOP.name());
    }

    void stopInMaster(WorkflowInstance workflowInstance) {
        try {
            final WorkflowInstanceStopResponse stopResponse = Clients
                    .withService(IWorkflowControlClient.class)
                    .withHost(workflowInstance.getHost())
                    .stopWorkflowInstance(new WorkflowInstanceStopRequest(workflowInstance.getId()));

            if (stopResponse != null && stopResponse.isSuccess()) {
                log.info("WorkflowInstance: {} stop success", workflowInstance.getName());
            } else {
                throw new ServiceException(
                        "WorkflowInstance: " + workflowInstance.getName() + " stop failed: " + stopResponse);
            }
        } catch (ServiceException e) {
            throw e;
        } catch (Exception e) {
            throw new ServiceException(
                    String.format("WorkflowInstance: %s stop failed", workflowInstance.getName()), e);
        }
    }

    public static class StopWorkflowInstanceOperation {

        private final StopWorkflowInstanceExecutorDelegate stopWorkflowInstanceExecutorDelegate;

        private WorkflowInstance workflowInstance;

        private User executeUser;

View on GitHub (pinned to 02eac45a1b)

Solutions

  1. Check whether the instance host still exists (master may have failed over); wait for failover to reassign or stop the instance on the new host
  2. Read the appended stopResponse to see the master-side failure reason
  3. Retry the stop once the master is stable
  4. If the master is permanently gone, force the instance to a final state via DB and clean up
  5. Check master logs for the stop/kill failure details

Example fix

// before
.withHost(workflowInstance.getHost())
    .stopWorkflowInstance(new WorkflowInstanceStopRequest(workflowInstance.getId()));
// after
// verify host still registered before RPC
if (!registryClient.serverExists(RegistryNodeType.MASTER, workflowInstance.getHost())) {
    throw new ServiceException("Host " + workflowInstance.getHost() + " no longer active; wait for failover");
}
Defensive patterns

Strategy: retry

Validate before calling

boolean hostAlive = registryClient.serverExists(RegistryNodeType.MASTER, instance.getHost());
if (!hostAlive) { throw new IllegalStateException("Instance host master is down; wait for failover"); }

Try / catch

try {
    delegate.execute(op);
} catch (ServiceException e) {
    log.error("Stop failed: {}", e.getMessage(), e);
    // check failover state, retry or force-finalize in DB
}

Prevention

When it happens

Trigger: Calling stopWorkflowInstance on the host at workflowInstance.getHost() and receiving a null response (master restarted/changed host) or a failed response (instance not found on that master, stop command rejected, task kill failed).

Common situations: Master failover after the instance host was recorded — the old host is gone so the RPC fails; instance already finished on the master side; master under heavy load timing out; kill operation failing on a non-killable task type.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/3f5f4f193330a2f7. Report an issue: GitHub.