{"record":{"id":"a39e25dc83ef137a","repo":"apache/dolphinscheduler","slug":"emr-task-submit-fail","errorCode":null,"errorMessage":"emr task submit fail","messagePattern":"emr task submit fail","errorType":"exception","errorClass":"TaskException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTask.java","lineNumber":93,"sourceCode":"    @Override\n    public void submitApplication() throws TaskException {\n        StepStatus stepStatus = null;\n        try {\n            AddJobFlowStepsRequest addJobFlowStepsRequest = createAddJobFlowStepsRequest();\n\n            // submit addJobFlowStepsRequest to aws\n            AddJobFlowStepsResult result = emrClient.addJobFlowSteps(addJobFlowStepsRequest);\n\n            clusterId = addJobFlowStepsRequest.getJobFlowId();\n            stepId = result.getStepIds().get(0);\n            // use clusterId-stepId as appIds\n            setAppIds(clusterId + TaskConstants.SUBTRACT_STRING + stepId);\n\n            stepStatus = getStepStatus();\n\n        } catch (EmrTaskException | SdkBaseException e) {\n            log.error(\"emr task submit failed with error\", e);\n            throw new TaskException(\"emr task submit fail\", e);\n        } finally {\n            final int exitStatusCode = calculateExitStatusCode(stepStatus);\n            setExitStatusCode(exitStatusCode);\n            log.info(\"emr task finished with step status : {}\", stepStatus);\n        }\n    }\n\n    @Override\n    public void trackApplicationStatus() throws TaskException {\n        StepStatus stepStatus = getStepStatus();\n\n        try {\n            while (waitingStateSet.contains(stepStatus.getState())) {\n                TimeUnit.SECONDS.sleep(10);\n                stepStatus = getStepStatus();\n            }\n        } catch (EmrTaskException | SdkBaseException e) {\n            log.error(\"emr task failed with error\", e);","sourceCodeStart":75,"sourceCodeEnd":111,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTask.java#L75-L111","documentation":"EmrAddStepsTask.submitApplication wraps EmrTaskException and AWS SDK SdkBaseException from the add-steps flow (runJobFlow for cluster creation and AddSteps) into TaskException 'emr task submit fail'. The finally block computes an exit status from the (possibly null) stepStatus. The real AWS error is in the cause and in the 'emr task submit failed with error' log line.","triggerScenarios":"runJobFlow or addJobFlowSteps throws SdkBaseException: invalid cluster/run-job-flow params, missing EMR IAM (service role/instance profile), S3 script path inaccessible, capacity errors, or the EMR API rejects the step configuration.","commonSituations":"EMR service role missing; bootstrap action or application name invalid for the releaseLabel; VPC/subnet lacks capacity (InsufficientInstanceCapacity); user lacks emr:RunJobFlow/emr:AddSteps; step jar/script path typo in S3.","solutions":["Read the 'caused by' AWS error in the log line 'emr task submit failed with error'","Verify IAM: task's execution credentials need emr:RunJobFlow/emr:AddSteps and the configured service role/instance profile exist","Confirm S3 paths for jars/scripts/bootstrap actions are correct and readable by the EMR instance profile","Check releaseLabel vs configured applications (e.g. Hive/Spark installed) and region","If InsufficientInstanceCapacity, retry later or change instance types/subnet"],"exampleFix":"// before\n\"steps\": [{ \"name\": \"s1\", \"jarPath\": \"s3://mybucket/missing.jar\" }] // path wrong -> step rejected\n// after\n\"steps\": [{ \"name\": \"s1\", \"jarPath\": \"s3://mybucket/lib/app-1.0.jar\", \"mainClass\": \"com.x.Main\" }]","handlingStrategy":"try-catch","validationCode":"// pre-flight: IAM + step config sanity before RunJobFlow/AddSteps\nAWSCredentials creds = new DefaultAWSCredentialsProviderChain().getCredentials();\n// caller-side check on task definition:\nif (StringUtils.isBlank(emrParameters.getClusterId()) && emrParameters.getSteps().isEmpty()) {\n    throw new IllegalArgumentException(\"Provide clusterId for existing cluster or steps for RunJobFlow\");\n}\n// verify S3 paths exist:\n// new AmazonS3Client().doesObjectExist(bucket, key) for jarPath/script paths","typeGuard":"boolean isServiceError(SdkBaseException e) {\n    return e instanceof AmazonServiceException\n        && ((AmazonServiceException) e).getErrorCode() != null;\n}","tryCatchPattern":"try {\n    submitApplication();\n} catch (TaskException e) {\n    Throwable cause = e.getCause();\n    if (cause instanceof AmazonServiceException) {\n        AmazonServiceException ase = (AmazonServiceException) cause;\n        log.error(\"EMR submit rejected: code={}, msg={}\", ase.getErrorCode(), ase.getErrorMessage());\n        if (\"InsufficientInstanceCapacity\".equals(ase.getErrorCode())) { /* retry later */ }\n    }\n    throw e;\n}","preventionTips":["Verify EMR service role and instance profile exist before submitting","Check S3 jar/script paths and EMR instance profile read permissions","Pre-check region capacity or configure flexible instance types/subnets","Align worker plugin AWS SDK version and credentials config with the master"],"tags":["aws","emr","submit","sdk"],"backgroundTag":"api-request-failed","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}