{"record":{"id":"8771b831e7af0c83","repo":"apache/dolphinscheduler","slug":"kubeflow-task-submit-command-failed","errorCode":null,"errorMessage":"Kubeflow task submit command failed","messagePattern":"Kubeflow task submit command failed","errorType":"exception","errorClass":"TaskException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-task-plugin/dolphinscheduler-task-kubeflow/src/main/java/org/apache/dolphinscheduler/plugin/kubeflow/KubeflowTask.java","lineNumber":123,"sourceCode":"\n    }\n\n    @Override\n    public void cancelApplication() throws TaskException {\n        String command = kubeflowHelper.buildDeleteCommand(yamlPath.toString());\n        log.info(\"Kubeflow task delete command: \\n{}\", command);\n        String message = runCommand(command);\n        log.info(\"Kubeflow task delete result: \\n{}\", message);\n        exitStatusCode = TaskConstants.EXIT_CODE_KILL;\n    }\n\n    protected String runCommand(String command) {\n        try {\n            exitStatusCode = TaskConstants.EXIT_CODE_SUCCESS;\n            return OSUtils.exeShell(new String[]{\"sh\", \"-c\", command});\n        } catch (Exception e) {\n            exitStatusCode = TaskConstants.EXIT_CODE_FAILURE;\n            throw new TaskException(\"Kubeflow task submit command failed\", e);\n        }\n    }\n\n    @Override\n    public List<String> getApplicationIds() throws TaskException {\n        return Collections.emptyList();\n    }\n\n    public void writeFiles() {\n        String yamlContent = kubeflowParameters.getYamlContent();\n        String clusterYAML = kubeflowParameters.getClusterYAML();\n\n        Map<String, Property> paramsMap = taskExecutionContext.getPrepareParamsMap();\n        yamlContent = ParameterUtils.convertParameterPlaceholders(yamlContent, ParameterUtils.convert(paramsMap));\n\n        yamlPath = Paths.get(taskExecutionContext.getExecutePath(), KubeflowHelper.CONSTANTS.YAML_FILE_PATH);\n        clusterYAMLPath =\n                Paths.get(taskExecutionContext.getExecutePath(), KubeflowHelper.CONSTANTS.CLUSTER_CONFIG_PATH);","sourceCodeStart":105,"sourceCodeEnd":141,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-task-plugin/dolphinscheduler-task-kubeflow/src/main/java/org/apache/dolphinscheduler/plugin/kubeflow/KubeflowTask.java#L105-L141","documentation":"KubeflowTask.runCommand executes kubectl-style commands via OSUtils.exeShell with 'sh -c'; any exception is wrapped in TaskException 'Kubeflow task submit command failed' and exitStatusCode set to failure. It is thrown while submitting (kubectl apply) or cancelling (kubectl delete) Kubeflow resources.","triggerScenarios":"runCommand(command) is called (e.g. 'kubectl apply -f <yaml>') and OSUtils.exeShell throws — kubectl binary missing, kubeconfig invalid/unreachable cluster, kubectl exiting with error surfaced as exception, or IO error spawning the shell.","commonSituations":"kubectl not installed on the worker, cluster unreachable/wrong API server URL, expired or missing service-account credentials, invalid resource YAML rejected by the API server, or network policy blocking egress to the cluster.","solutions":["Run the exact command from worker logs manually with 'sh -c' to see kubectl's stderr.","Verify kubectl is installed and on PATH for the worker user.","Validate kubeconfig / cluster connectivity: 'kubectl cluster-info' with the same kubeconfig.","Check the generated YAML is valid and the target namespace/CRDs exist.","Confirm network access from worker to the K8s API server (firewall/DNS)."],"exampleFix":"// before\n} catch (Exception e) {\n    throw new TaskException(\"Kubeflow task submit command failed\", e);\n}\n// after\n} catch (Exception e) {\n    log.error(\"kubeflow command failed: {}\", command, e);\n    throw new TaskException(\"Kubeflow task submit command failed: \" + command + \" -> \" + e.getMessage(), e);\n}","handlingStrategy":"try-catch","validationCode":"// preflight cluster access with the same command path\nint rc = Runtime.getRuntime().exec(new String[]{\"sh\",\"-c\",\"command -v kubectl && kubectl cluster-info\"}).waitFor();\nif (rc != 0) throw new IllegalStateException(\"kubectl missing or cluster unreachable\");","typeGuard":"static boolean isSubmitCommandFailure(TaskException e) {\n    return e.getMessage() != null && e.getMessage().startsWith(\"Kubeflow task submit command failed\");\n}","tryCatchPattern":"try {\n    String out = kubeflowTask.runCommand(\"kubectl apply -f \" + yamlPath);\n} catch (TaskException e) {\n    log.error(\"kubectl command failed; check cluster connectivity, kubeconfig and YAML validity\", e.getCause());\n    if (isTransientNetwork(e.getCause())) retryWithBackoff();\n}","preventionTips":["Install kubectl on worker images and keep kubeconfig/service accounts valid.","Run 'kubectl cluster-info' and 'kubectl auth can-i' as a deployment smoke test.","Validate resource YAML with 'kubectl apply --dry-run=client' before scheduling.","Ensure network egress from workers to the K8s API server (DNS, firewall)."],"tags":["kubeflow","kubectl","shell-execution","k8s"],"backgroundTag":"command-execution-failed","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}