{"record":{"id":"0c7e9bb7bea0fce5","repo":"apache/dolphinscheduler","slug":"generate-spark-sql-script-error","errorCode":null,"errorMessage":"generate spark sql script error","messagePattern":"generate spark sql script error","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java","lineNumber":279,"sourceCode":"            String script = replaceParam(sqlContent);\n\n            log.info(\"raw script : {}\", script);\n            log.info(\"task execute path : {}\", taskExecutionContext.getExecutePath());\n\n            Set<PosixFilePermission> perms = PosixFilePermissions.fromString(RWXR_XR_X);\n            FileAttribute<Set<PosixFilePermission>> attr = PosixFilePermissions.asFileAttribute(perms);\n            try {\n                if (SystemUtils.IS_OS_WINDOWS) {\n                    Files.createFile(path);\n                } else {\n                    if (!file.getParentFile().exists()) {\n                        file.getParentFile().mkdirs();\n                    }\n                    Files.createFile(path, attr);\n                }\n                Files.write(path, script.getBytes(), StandardOpenOption.APPEND);\n            } catch (IOException e) {\n                throw new RuntimeException(\"generate spark sql script error\", e);\n            }\n\n        }\n        return scriptFileName;\n    }\n\n    private String replaceParam(String script) {\n        script = script.replaceAll(\"\\\\r\\\\n\", System.lineSeparator());\n        // replace placeholder\n        Map<String, Property> paramsMap = taskExecutionContext.getPrepareParamsMap();\n        script = ParameterUtils.convertParameterPlaceholders(script, ParameterUtils.convert(paramsMap));\n        return script;\n    }\n\n    @Override\n    public AbstractParameters getParameters() {\n        return sparkParameters;\n    }","sourceCodeStart":261,"sourceCodeEnd":297,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java#L261-L297","documentation":"SparkTask.generateScriptFile(), called from populateSparkOptions, writes the SQL content to a generated script file under the working directory (Files.createFile + Files.write with APPEND). Any IOException while creating or writing that file is wrapped as RuntimeException(\"generate spark sql script error\"). The script file path is what the spark-submit command later consumes.","triggerScenarios":"Files.createFile throws FileAlreadyExistsException, AccessDeniedException, or NoSuchFileException while writing the generated .sql script — i.e. the target file exists from a prior run, the directory lacks write permission, or the parent dir path is invalid.","commonSituations":"Disk full on the worker node; leftover script file from a previous crashed run with a conflicting name; worker OS user lacking write access to the task working directory; tenant work dir removed while the task instance is running.","solutions":["Check free disk space on the worker (df -h) and clean old task working directories under the tenant's exec dir","Inspect the wrapped IOException: delete the stale script file or fix permissions (chown worker user, chmod u+w) on the target directory","Re-run the task; if FileAlreadyExists persists, enable task-instance-level working dirs or clean the instance work dir between retries"],"exampleFix":"// before: work dir owned by root\nls -l /tmp/dolphinscheduler/exec/tenant  -> root root\n// after\nchown -R dolphinscheduler:dolphinscheduler /tmp/dolphinscheduler/exec/tenant","handlingStrategy":"try-catch","validationCode":"File workDir = taskWorkingDirectory.toFile();\nif (!workDir.canWrite()) throw new IllegalStateException(\"work dir not writable: \" + workDir);\nif (workDir.getFreeSpace() < MIN_FREE_BYTES) throw new IllegalStateException(\"low disk\");","typeGuard":null,"tryCatchPattern":"try {\n    sparkTask.handle(callBack);\n} catch (RuntimeException e) {\n    if (e.getCause() instanceof FileAlreadyExistsException) {\n        // clean stale generated script and retry\n    }\n}","preventionTips":["Monitor disk space and clean old task working directories on workers","Ensure tenant work dirs are owned/writable by the worker OS user","Use per-instance working directories so generated script names never collide"],"tags":["file-write","spark","io"],"backgroundTag":"file-write-failed","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}