apache/dolphinscheduler · error · TaskException

read sql content error

Error message

read sql content error

What it means

SparkTask.populateSparkOptions(), reached via getScript(), reads the user's SQL resource file from local storage with FileUtils.readFileToString(..., UTF_8) when the spark task is a SQL-type job running a stored script. An IOException during that read is rethrown as TaskException("read sql content error"). The Spark job never starts; the failure is purely in loading the SQL source.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java:216

            String sqlContent = "";
            String resourceFileName = "";
            args.add(SparkConstants.SQL_FROM_FILE);
            if (SparkConstants.TYPE_FILE.equals(sparkParameters.getSqlExecutionType())) {
                final List<ResourceInfo> resourceInfos = sparkParameters.getResourceList();
                if (resourceInfos.size() > 1) {
                    log.warn("more than 1 files detected, use the first one by default");
                }

                try {
                    resourceFileName = resourceInfos.get(0).getResourceName();
                    ResourceContext resourceContext = taskExecutionContext.getResourceContext();
                    sqlContent = FileUtils.readFileToString(
                            new File(
                                    resourceContext.getResourceItem(resourceFileName).getResourceAbsolutePathInLocal()),
                            StandardCharsets.UTF_8);
                } catch (IOException e) {
                    log.error("read sql content from file {} error ", resourceFileName, e);
                    throw new TaskException("read sql content error", e);
                }
            } else {
                sqlContent = sparkParameters.getRawScript();
            }
            args.add(generateScriptFile(sqlContent));
        }
        return args;
    }

    private void populateSparkResourceDefinitions(List<String> args) {
        int driverCores = sparkParameters.getDriverCores();
        if (driverCores > 0) {
            args.add(String.format(SparkConstants.DRIVER_CORES, driverCores));
        }

        String driverMemory = sparkParameters.getDriverMemory();
        if (StringUtils.isNotEmpty(driverMemory)) {
            args.add(String.format(SparkConstants.DRIVER_MEMORY, driverMemory));

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Verify the resource file still exists in the resource center and re-upload/re-select it if it was renamed or deleted, then re-save the task
2. Check the local resource path in the wrapped IOException and fix OS permissions so the worker OS user can read it
3. Confirm the worker's shared-storage mount (HDFS/S3 cache dir) is healthy; re-download the resource or restart the worker if the cache is stale

Example fix

// before: task references deleted resource
resourceFileName = "query.sql"  // deleted from resource center
// after: re-upload and re-select
// Resource Center -> upload query.sql -> Spark SQL task -> Resource: query.sql -> save

File f = new File(resourceContext.getResourceItem(resourceFileName).getResourceAbsolutePathInLocal());
if (!f.exists() || !f.canRead()) {
    throw new IllegalStateException("SQL resource missing/unreadable: " + f);
}

boolean resourceReadable(ResourceContext ctx, String name) {
    try {
        File f = new File(ctx.getResourceItem(name).getResourceAbsolutePathInLocal());
        return f.isFile() && f.canRead();
    } catch (Exception e) { return false; }
}

try {
    sparkTask.handle(callBack);
} catch (TaskException e) {
    // re-fetch/re-upload the resource, then retry once
}

When it happens

Trigger: The resource item returned by resourceContext.getResourceItem(resourceFileName) resolves to a local path that does not exist, was deleted by tenant-dir cleanup, or is unreadable due to permissions; the resource was moved/renamed in the resource center after the task definition referenced it.

Common situations: Resource center file deleted or renamed while the workflow still references its ID; worker runs as a user without read permission on the tenant resource directory; shared-storage (HDFS/S3) mount broken on the worker so the downloaded resource is absent locally.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Can not find valid resource by name %s
• 20016
• ILLEGAL_RESOURCE_PATH
• The resource path is null
• Invalidated resource path: ${resourceAbsolutePath}

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/37eb6041f24e4aa8. Report an issue: GitHub.