apache/dolphinscheduler · error · EmrTaskException

emr task params is not valid

Error message

emr task params is not valid

What it means

AbstractEmrTask.init parses taskParams into EmrParameters and validates them via checkParameters(). If parsing yields null or validation fails, it throws EmrTaskException 'emr task params is not valid' before any AWS call is made — the task definition itself is incomplete.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/AbstractEmrTask.java:84

            .setPropertyNamingStrategy(new PropertyNamingStrategy.UpperCamelCaseStrategy());

    /**
     * constructor
     *
     * @param taskExecutionContext taskExecutionContext
     */
    protected AbstractEmrTask(TaskExecutionContext taskExecutionContext) {
        super(taskExecutionContext);
        this.taskExecutionContext = taskExecutionContext;
    }

    @Override
    public void init() {
        final String taskParams = taskExecutionContext.getTaskParams();
        emrParameters = JSONUtils.parseObject(taskParams, EmrParameters.class);
        log.info("Initialize emr task params:{}", JSONUtils.toPrettyJsonString(taskParams));
        if (emrParameters == null || !emrParameters.checkParameters()) {
            throw new EmrTaskException("emr task params is not valid");
        }
        emrClient = createEmrClient();
    }

    @Override
    public AbstractParameters getParameters() {
        return emrParameters;
    }

    protected AmazonElasticMapReduce createEmrClient() {
        Map<String, String> awsProperties = PropertyUtils.getByPrefix("aws.emr.", "");
        return AmazonElasticMapReduceClientFactory.createAmazonElasticMapReduceClient(awsProperties);
    }
}

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Open the task definition JSON and confirm required EmrParameters fields are present (clusterId, steps for add-steps tasks)
2. Re-save the task in the UI so the plugin generates a complete params object
3. Upgrade/align worker plugin version with the API/UI version to avoid schema skew
4. Log the raw taskParams (init already logs it) and diff against a known-good definition

Example fix

// before
taskParams = "{\"localParams\":[]}" // no steps, no clusterId
// after
taskParams = "{\"localParams\":[],\"programType\":\"SPARK\",\"clusterId\":\"j-XXXX\",\"steps\":[{\"name\":\"step1\",\"actionOnFailure\":\"CONTINUE\",\"mainClass\":\"...\",\"jarPath\":\"s3://b/app.jar\"}]}"

// before creating the task, verify required fields in taskParams JSON
Map<String,Object> params = JSONUtils.parseObject(rawParams, Map.class);
List<String> required = List.of("clusterId", "steps"); // for EmrAddStepsTask
for (String k : required) {
    if (params == null || !params.containsKey(k))
        throw new IllegalArgumentException("taskParams missing required field: " + k);
}

boolean hasValidEmrParams(String taskParams) {
    EmrParameters p = JSONUtils.parseObject(taskParams, EmrParameters.class);
    return p != null && p.checkParameters();
}

try {
    task.init();
} catch (EmrTaskException e) {
    if (e.getMessage().contains("emr task params is not valid")) {
        log.error("Fix task definition params; raw: {}", taskExecutionContext.getTaskParams(), e);
    }
    throw e;
}

When it happens

Trigger: JSONUtils.parseObject returns null (taskParams blank or not valid JSON), or EmrParameters.checkParameters() returns false because required fields are missing — for EmrAddStepsTask: steps/clusterId absent; for EmrJobDriverTask/EmrClusterTask: their required fields (releaseLabel, jobFlowId, etc.) absent.

Common situations: Task created via older UI/plugin version before a new required field existed (version skew); user left 'steps' or cluster ID empty; hand-edited task JSON dropped a mandatory property; wrong task type mapping deserializes params into EmrParameters incorrectly.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• EMR Serverless task params are not valid
• dvc task params is not valid
• emr task submit fail
• Execute emr task failed
• flink task params is not valid

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/fd4a4685d1d6af78. Report an issue: GitHub.