apache/dolphinscheduler · error · RuntimeException

spark task params is not valid

Error message

spark task params is not valid

What it means

SparkTask.init() validates the parsed SparkParameters; if checkParameters() returns false it throws RuntimeException("spark task params is not valid"). Unlike ShellTask, the null-params case only logs and returns — this exception specifically means params parsed but required fields (main class, main jar/python file, deploy mode) are missing. Note it is a raw RuntimeException, not TaskException.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java:83

    private final TaskExecutionContext taskExecutionContext;

    public SparkTask(TaskExecutionContext taskExecutionContext) {
        super(taskExecutionContext);
        this.taskExecutionContext = taskExecutionContext;
    }

    @Override
    public void init() {

        sparkParameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), SparkParameters.class);

        if (null == sparkParameters) {
            log.error("Spark params is null");
            return;
        }

        if (!sparkParameters.checkParameters()) {
            throw new RuntimeException("spark task params is not valid");
        }

        log.info("Initialize spark task params {}", JSONUtils.toPrettyJsonString(sparkParameters));
    }

    @Override
    protected String getScript() {
        /**
         * (1) spark-submit [options] <app jar | python file> [app arguments]
         * (2) spark-sql [options] -f <filename>
         */
        List<String> args = new ArrayList<>();

        String sparkCommand;
        // If the programType is SQL, execute bin/spark-sql
        if (sparkParameters.getProgramType() == ProgramType.SQL) {
            sparkCommand = SparkConstants.SPARK_SQL_COMMAND;
        } else {

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Open the Spark task and set all required fields: mainJar (or main python file), mainClass for Java/Scala, and a valid deployMode
2. Pre-validate params with sparkParameters.checkParameters() before creating the task instance
3. If hitting this on an existing workflow after an upgrade, re-open and re-save the task node to refresh the params JSON schema

Example fix

// before
{"taskType":"SPARK","taskParams":{"deployMode":"client"}}
// after
{"taskType":"SPARK","taskParams":{"mainClass":"com.example.Main","mainJar":"hdfs:///apps/job.jar","deployMode":"client","programType":"JAVA"}}

SparkParameters sp = JSONUtils.parseObject(taskParams, SparkParameters.class);
if (sp == null) return; // null only logs in SparkTask.init
if (!sp.checkParameters()) {
    throw new IllegalArgumentException("spark params require mainJar/mainClass and valid deployMode");
}

boolean isValidSparkParams(String json) {
    SparkParameters p = JSONUtils.parseObject(json, SparkParameters.class);
    return p == null || p.checkParameters(); // treat null as early-return path
}

try {
    sparkTask.init();
} catch (RuntimeException e) {
    // checkParameters failed: fix mainClass/mainJar/deployMode in taskParams
}

When it happens

Trigger: SparkParameters.checkParameters() returns false — typically mainClass missing for Java/Scala jobs, or mainJar empty, or invalid deployMode; task params JSON parses fine but omits these required fields.

Common situations: Spark task saved without selecting the main jar; Python-type spark task whose mainJar path is blank; switching task type in the UI left stale empty spark params; API-submitted taskDef missing mainClass.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• dvc task params is not valid
• EMR Serverless task params are not valid
• emr task params is not valid
• flink task params is not valid
• flink task params is not valid

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/394ddf19cb2228c9. Report an issue: GitHub.