apache/dolphinscheduler · error · TaskException

emr task submit failed

Error message

emr task submit failed

What it means

A catch-all in EmrJobFlowTask.submitApplication: creating the RunJobFlowRequest (bad jobFlowDefineJson) or calling emrClient.runJobFlow threw — invalid AWS credentials, insufficient IAM permission for elasticmapreduce:RunJobFlow, bad service role/instance profile, or an SDK error. The job flow was never started; clusterId remains unset and the task fails.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java:86

    @Override
    public void submitApplication() throws TaskException {
        ClusterStatus clusterStatus = null;
        try {
            RunJobFlowRequest runJobFlowRequest = createRunJobFlowRequest();

            // submit runJobFlowRequest to aws
            RunJobFlowResult result = emrClient.runJobFlow(runJobFlowRequest);

            clusterId = result.getJobFlowId();
            // Failover on EMR Task type has not been implemented. In this time, DS only supports failover on yarn task
            // type . Other task type, such as EMR task, k8s task not ready yet.
            setAppIds(clusterId);

            clusterStatus = getClusterStatus();

        } catch (EmrTaskException | SdkBaseException e) {
            log.error("emr task submit failed with error", e);
            throw new TaskException("emr task submit failed", e);
        } finally {
            final int exitStatusCode = calculateExitStatusCode(clusterStatus);
            setExitStatusCode(exitStatusCode);
            log.info("emr task finished with cluster status : {}", clusterStatus);
        }
    }

    @Override
    public void trackApplicationStatus() throws TaskException {
        ClusterStatus clusterStatus = null;
        try {
            clusterStatus = getClusterStatus();

            while (waitingStateSet.contains(clusterStatus.getState())) {
                TimeUnit.SECONDS.sleep(10);
                clusterStatus = getClusterStatus();
            }
        } catch (EmrTaskException | SdkBaseException e) {

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Inspect the caused-by exception: JsonProcessingException means bad jobFlowDefineJson; SdkBaseException means AWS-side failure
2. Fix the AWS credential/policy configuration (RunJobFlow, PassRole permissions, service role and instance profile ARNs)
3. Validate jobFlowDefineJson against the RunJobFlowRequest schema before running
4. Retry after transient AWS errors

When it happens

Trigger: Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java:86 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/cc3617ebf554e153. Report an issue: GitHub.