apache/dolphinscheduler · error · AliyunServerlessSparkTaskException

Failed to start job run!

Error message

Failed to start job run! 

What it means

In handle(), after building a StartJobRunRequest the task calls aliyunServerlessSparkClient.startJobRun inside RetryUtils.retryFunction; any exception is wrapped as AliyunServerlessSparkTaskException "Failed to start job run!" and the task fails after retries are exhausted.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-aliyunserverlessspark/src/main/java/org/apache/dolphinscheduler/plugin/task/aliyunserverlessspark/AliyunServerlessSparkTask.java:157

        if (getTemplateResponse != null) {
            templateConf = getTemplateResponse.getBody()
                    .getData()
                    .getSparkConf()
                    .stream()
                    .map(item -> "--conf " + item.getKey() + "=" + item.getValue())
                    .collect(Collectors.joining(" "));

            templateDisplayReleaseVersion = getTemplateResponse.getBody().getData().getDisplaySparkVersion();
            templateFusion = getTemplateResponse.getBody().getData().getFusion();
        }

        StartJobRunRequest startJobRunRequest = buildStartJobRunRequest(aliyunServerlessSparkParameters);
        StartJobRunResponse startJobRunResponse = RetryUtils.retryFunction(() -> {
            try {
                return aliyunServerlessSparkClient.startJobRun(
                        aliyunServerlessSparkParameters.getWorkspaceId(), startJobRunRequest);
            } catch (Exception e) {
                throw new AliyunServerlessSparkTaskException("Failed to start job run! ", e);
            }
        }, retryPolicy);

        jobRunId = startJobRunResponse.getBody().getJobRunId();
        setAppIds(jobRunId);
        log.info("Successfully submitted serverless spark job, jobRunId - {}", jobRunId);

        while (!RunState.isFinal(currentState)) {
            GetJobRunRequest getJobRunRequest = buildGetJobRunRequest();

            GetJobRunResponse getJobRunResponse = RetryUtils.retryFunction(() -> {
                try {
                    return aliyunServerlessSparkClient
                            .getJobRun(aliyunServerlessSparkParameters.getWorkspaceId(), jobRunId,
                                    getJobRunRequest);
                } catch (Exception e) {
                    throw new AliyunServerlessSparkTaskException("Failed to get job run!", e);
                }

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Check the wrapped cause/logs to identify the Aliyun API error code (auth vs quota vs validation)
2. Validate the job template configuration and parameters used in buildStartJobRunRequest
3. Verify credentials/permissions for emr-serverless-spark:StartJobRun
4. Ensure workspace has available quota and the region/endpoint are correct; retry later if throttled

// Validate request before submitting
if (startJobRunRequest == null || workspaceId == null || workspaceId.isEmpty()) {
    throw new IllegalArgumentException("workspaceId and StartJobRunRequest must be valid before submit");
}

try {
    task.handle();
} catch (AliyunServerlessSparkTaskException e) {
    if (e.getMessage().contains("Failed to start job run")) {
        log.error("Job submission failed; check Aliyun error code in cause", e);
        // fail the task execution status; do not blind-retry validation errors
    }
}

When it happens

Trigger: startJobRun API call failing due to invalid job configuration (SQL/file/parameters), insufficient quota or permissions in the workspace, throttling, or transient network errors persisting beyond the retryPolicy.

Common situations: Invalid Spark job template parameters or bad SQL; workspace quota exhausted; RAM role lacking StartJobRun permission; region/endpoint mismatch; Aliyun service-side errors (4xx/5xx) under load.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• Failed to get template info
• Aliyun-Serverless-Spark task parameters are not valid!
• Failed to build Aliyun-Serverless-Spark client!
• Failed to get job run!
• Failed to cancel job run!

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/025aa808bff58547. Report an issue: GitHub.