apache/dolphinscheduler · error · TaskException

Kubeflow task submit command failed

Error message

Kubeflow task submit command failed

What it means

KubeflowTask.runCommand executes kubectl-style commands via OSUtils.exeShell with 'sh -c'; any exception is wrapped in TaskException 'Kubeflow task submit command failed' and exitStatusCode set to failure. It is thrown while submitting (kubectl apply) or cancelling (kubectl delete) Kubeflow resources.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-kubeflow/src/main/java/org/apache/dolphinscheduler/plugin/kubeflow/KubeflowTask.java:123

    }

    @Override
    public void cancelApplication() throws TaskException {
        String command = kubeflowHelper.buildDeleteCommand(yamlPath.toString());
        log.info("Kubeflow task delete command: \n{}", command);
        String message = runCommand(command);
        log.info("Kubeflow task delete result: \n{}", message);
        exitStatusCode = TaskConstants.EXIT_CODE_KILL;
    }

    protected String runCommand(String command) {
        try {
            exitStatusCode = TaskConstants.EXIT_CODE_SUCCESS;
            return OSUtils.exeShell(new String[]{"sh", "-c", command});
        } catch (Exception e) {
            exitStatusCode = TaskConstants.EXIT_CODE_FAILURE;
            throw new TaskException("Kubeflow task submit command failed", e);
        }
    }

    @Override
    public List<String> getApplicationIds() throws TaskException {
        return Collections.emptyList();
    }

    public void writeFiles() {
        String yamlContent = kubeflowParameters.getYamlContent();
        String clusterYAML = kubeflowParameters.getClusterYAML();

        Map<String, Property> paramsMap = taskExecutionContext.getPrepareParamsMap();
        yamlContent = ParameterUtils.convertParameterPlaceholders(yamlContent, ParameterUtils.convert(paramsMap));

        yamlPath = Paths.get(taskExecutionContext.getExecutePath(), KubeflowHelper.CONSTANTS.YAML_FILE_PATH);
        clusterYAMLPath =
                Paths.get(taskExecutionContext.getExecutePath(), KubeflowHelper.CONSTANTS.CLUSTER_CONFIG_PATH);

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Run the exact command from worker logs manually with 'sh -c' to see kubectl's stderr.
2. Verify kubectl is installed and on PATH for the worker user.
3. Validate kubeconfig / cluster connectivity: 'kubectl cluster-info' with the same kubeconfig.
4. Check the generated YAML is valid and the target namespace/CRDs exist.
5. Confirm network access from worker to the K8s API server (firewall/DNS).

Example fix

// before
} catch (Exception e) {
    throw new TaskException("Kubeflow task submit command failed", e);
}
// after
} catch (Exception e) {
    log.error("kubeflow command failed: {}", command, e);
    throw new TaskException("Kubeflow task submit command failed: " + command + " -> " + e.getMessage(), e);
}

// preflight cluster access with the same command path
int rc = Runtime.getRuntime().exec(new String[]{"sh","-c","command -v kubectl && kubectl cluster-info"}).waitFor();
if (rc != 0) throw new IllegalStateException("kubectl missing or cluster unreachable");

static boolean isSubmitCommandFailure(TaskException e) {
    return e.getMessage() != null && e.getMessage().startsWith("Kubeflow task submit command failed");
}

try {
    String out = kubeflowTask.runCommand("kubectl apply -f " + yamlPath);
} catch (TaskException e) {
    log.error("kubectl command failed; check cluster connectivity, kubeconfig and YAML validity", e.getCause());
    if (isTransientNetwork(e.getCause())) retryWithBackoff();
}

When it happens

Trigger: runCommand(command) is called (e.g. 'kubectl apply -f <yaml>') and OSUtils.exeShell throws — kubectl binary missing, kubeconfig invalid/unreachable cluster, kubectl exiting with error surfaced as exception, or IO error spawning the shell.

Common situations: kubectl not installed on the worker, cluster unreachable/wrong API server URL, expired or missing service-account credentials, invalid resource YAML rejected by the API server, or network policy blocking egress to the cluster.

Related errors

• Kubeflow task params is not valid
• Kubeflow task write yaml file failed
• 120034
• 1300006
• USER_NO_OPERATION_PERM

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/8771b831e7af0c83. Report an issue: GitHub.