apache/dolphinscheduler · error · EmrTaskException

fetch cluster status failed

Error message

fetch cluster status failed

What it means

getClusterStatus caught an AWS SDK exception while calling emrClient.describeCluster (or got a null result), so the EMR cluster's state could not be fetched for exit-code mapping. Typical causes: expired/invalid credentials, throttling, the cluster terminated and deleted outside DS, or network failure. The sentinel message 'fetch cluster status failed' wraps the SDK detail in the cause.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java:173

                    String code = stateChangeReason.getCode();
                    if (code != null
                            && code.equalsIgnoreCase(ClusterStateChangeReasonCode.ALL_STEPS_COMPLETED.toString())) {
                        return TaskConstants.EXIT_CODE_SUCCESS;
                    } else {
                        return TaskConstants.EXIT_CODE_KILL;
                    }
                default:
                    return TaskConstants.EXIT_CODE_FAILURE;
            }
        }

    }

    private ClusterStatus getClusterStatus() {
        DescribeClusterRequest describeClusterRequest = new DescribeClusterRequest().withClusterId(clusterId);
        DescribeClusterResult result = emrClient.describeCluster(describeClusterRequest);
        if (result == null) {
            throw new EmrTaskException("fetch cluster status failed");
        }
        ClusterStatus clusterStatus = result.getCluster().getStatus();
        log.info("emr cluster [clusterId:{}] running with status:{}", clusterId, clusterStatus);
        return clusterStatus;

    }

    @Override
    public void cancelApplication() throws TaskException {
        log.info("trying terminate job flow, taskId:{}, clusterId:{}", this.taskExecutionContext.getTaskInstanceId(),
                clusterId);
        TerminateJobFlowsRequest terminateJobFlowsRequest = new TerminateJobFlowsRequest().withJobFlowIds(clusterId);
        TerminateJobFlowsResult terminateJobFlowsResult = emrClient.terminateJobFlows(terminateJobFlowsRequest);
        log.info("the result of terminate job flow is:{}", terminateJobFlowsResult);
    }

}

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Inspect the caused-by SdkBaseException for auth/throttling/not-found detail
2. Verify the cluster (jobFlowId) still exists in AWS — externally deleted clusters make describeCluster fail
3. Retry status polling after transient throttling or network errors
4. Check the task's AWS credentials and region configuration

When it happens

Trigger: Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java:173 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/73df081e8f6b013f. Report an issue: GitHub.