apache/dolphinscheduler · error · RuntimeException

generate spark sql script error

Error message

generate spark sql script error

What it means

SparkTask.generateScriptFile(), called from populateSparkOptions, writes the SQL content to a generated script file under the working directory (Files.createFile + Files.write with APPEND). Any IOException while creating or writing that file is wrapped as RuntimeException("generate spark sql script error"). The script file path is what the spark-submit command later consumes.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java:279

            String script = replaceParam(sqlContent);

            log.info("raw script : {}", script);
            log.info("task execute path : {}", taskExecutionContext.getExecutePath());

            Set<PosixFilePermission> perms = PosixFilePermissions.fromString(RWXR_XR_X);
            FileAttribute<Set<PosixFilePermission>> attr = PosixFilePermissions.asFileAttribute(perms);
            try {
                if (SystemUtils.IS_OS_WINDOWS) {
                    Files.createFile(path);
                } else {
                    if (!file.getParentFile().exists()) {
                        file.getParentFile().mkdirs();
                    }
                    Files.createFile(path, attr);
                }
                Files.write(path, script.getBytes(), StandardOpenOption.APPEND);
            } catch (IOException e) {
                throw new RuntimeException("generate spark sql script error", e);
            }

        }
        return scriptFileName;
    }

    private String replaceParam(String script) {
        script = script.replaceAll("\\r\\n", System.lineSeparator());
        // replace placeholder
        Map<String, Property> paramsMap = taskExecutionContext.getPrepareParamsMap();
        script = ParameterUtils.convertParameterPlaceholders(script, ParameterUtils.convert(paramsMap));
        return script;
    }

    @Override
    public AbstractParameters getParameters() {
        return sparkParameters;
    }

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Check free disk space on the worker (df -h) and clean old task working directories under the tenant's exec dir
2. Inspect the wrapped IOException: delete the stale script file or fix permissions (chown worker user, chmod u+w) on the target directory
3. Re-run the task; if FileAlreadyExists persists, enable task-instance-level working dirs or clean the instance work dir between retries

Example fix

// before: work dir owned by root
ls -l /tmp/dolphinscheduler/exec/tenant  -> root root
// after
chown -R dolphinscheduler:dolphinscheduler /tmp/dolphinscheduler/exec/tenant

File workDir = taskWorkingDirectory.toFile();
if (!workDir.canWrite()) throw new IllegalStateException("work dir not writable: " + workDir);
if (workDir.getFreeSpace() < MIN_FREE_BYTES) throw new IllegalStateException("low disk");

try {
    sparkTask.handle(callBack);
} catch (RuntimeException e) {
    if (e.getCause() instanceof FileAlreadyExistsException) {
        // clean stale generated script and retry
    }
}

When it happens

Trigger: Files.createFile throws FileAlreadyExistsException, AccessDeniedException, or NoSuchFileException while writing the generated .sql script — i.e. the target file exists from a prior run, the directory lacks write permission, or the parent dir path is invalid.

Common situations: Disk full on the worker node; leftover script file from a previous crashed run with a conflicting name; worker OS user lacking write access to the task working directory; tenant work dir removed while the task instance is running.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Kubeflow task write yaml file failed
• Create xlsx directory error
• generate excel error
• Update the resource file from content: {fileAbsolutePath} fa
• Download the resource file: {fileAbsolutePath} failed

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/0c7e9bb7bea0fce5. Report an issue: GitHub.