{"record":{"id":"394ddf19cb2228c9","repo":"apache/dolphinscheduler","slug":"spark-task-params-is-not-valid","errorCode":null,"errorMessage":"spark task params is not valid","messagePattern":"spark task params is not valid","errorType":"validation","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java","lineNumber":83,"sourceCode":"    private final TaskExecutionContext taskExecutionContext;\n\n    public SparkTask(TaskExecutionContext taskExecutionContext) {\n        super(taskExecutionContext);\n        this.taskExecutionContext = taskExecutionContext;\n    }\n\n    @Override\n    public void init() {\n\n        sparkParameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), SparkParameters.class);\n\n        if (null == sparkParameters) {\n            log.error(\"Spark params is null\");\n            return;\n        }\n\n        if (!sparkParameters.checkParameters()) {\n            throw new RuntimeException(\"spark task params is not valid\");\n        }\n\n        log.info(\"Initialize spark task params {}\", JSONUtils.toPrettyJsonString(sparkParameters));\n    }\n\n    @Override\n    protected String getScript() {\n        /**\n         * (1) spark-submit [options] <app jar | python file> [app arguments]\n         * (2) spark-sql [options] -f <filename>\n         */\n        List<String> args = new ArrayList<>();\n\n        String sparkCommand;\n        // If the programType is SQL, execute bin/spark-sql\n        if (sparkParameters.getProgramType() == ProgramType.SQL) {\n            sparkCommand = SparkConstants.SPARK_SQL_COMMAND;\n        } else {","sourceCodeStart":65,"sourceCodeEnd":101,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java#L65-L101","documentation":"SparkTask.init() validates the parsed SparkParameters; if checkParameters() returns false it throws RuntimeException(\"spark task params is not valid\"). Unlike ShellTask, the null-params case only logs and returns — this exception specifically means params parsed but required fields (main class, main jar/python file, deploy mode) are missing. Note it is a raw RuntimeException, not TaskException.","triggerScenarios":"SparkParameters.checkParameters() returns false — typically mainClass missing for Java/Scala jobs, or mainJar empty, or invalid deployMode; task params JSON parses fine but omits these required fields.","commonSituations":"Spark task saved without selecting the main jar; Python-type spark task whose mainJar path is blank; switching task type in the UI left stale empty spark params; API-submitted taskDef missing mainClass.","solutions":["Open the Spark task and set all required fields: mainJar (or main python file), mainClass for Java/Scala, and a valid deployMode","Pre-validate params with sparkParameters.checkParameters() before creating the task instance","If hitting this on an existing workflow after an upgrade, re-open and re-save the task node to refresh the params JSON schema"],"exampleFix":"// before\n{\"taskType\":\"SPARK\",\"taskParams\":{\"deployMode\":\"client\"}}\n// after\n{\"taskType\":\"SPARK\",\"taskParams\":{\"mainClass\":\"com.example.Main\",\"mainJar\":\"hdfs:///apps/job.jar\",\"deployMode\":\"client\",\"programType\":\"JAVA\"}}","handlingStrategy":"validation","validationCode":"SparkParameters sp = JSONUtils.parseObject(taskParams, SparkParameters.class);\nif (sp == null) return; // null only logs in SparkTask.init\nif (!sp.checkParameters()) {\n    throw new IllegalArgumentException(\"spark params require mainJar/mainClass and valid deployMode\");\n}","typeGuard":"boolean isValidSparkParams(String json) {\n    SparkParameters p = JSONUtils.parseObject(json, SparkParameters.class);\n    return p == null || p.checkParameters(); // treat null as early-return path\n}","tryCatchPattern":"try {\n    sparkTask.init();\n} catch (RuntimeException e) {\n    // checkParameters failed: fix mainClass/mainJar/deployMode in taskParams\n}","preventionTips":["Always set mainJar and mainClass (Java/Scala) or a main python file in Spark task params","Validate checkParameters() in workflow CI before submission","Re-save task nodes after upgrading or changing task-type templates"],"tags":["spark","validation","task-params"],"backgroundTag":"schema-validation-failed","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}