apache/dolphinscheduler · error · TaskException

Execute jupyter task failed

Error message

Execute jupyter task failed

What it means

JupyterTask.handle() wraps any non-interruption failure of the jupyter command execution into TaskException 'Execute jupyter task failed'. It is the generic catch-all when the jupyter/conda/pip shell process fails to run or exits with an error; exitStatusCode is set to -1.

Source

Thrown at dolphinscheduler-task-plugin/dolphinscheduler-task-jupyter/src/main/java/org/apache/dolphinscheduler/plugin/task/jupyter/JupyterTask.java:101

    public void handle(TaskCallBack taskCallBack) throws TaskException {
        try {
            IShellInterceptorBuilder<?, ?> shellActuatorBuilder = ShellInterceptorBuilderFactory.newBuilder()
                    .properties(ParameterUtils.convert(taskRequest.getPrepareParamsMap()))
                    .appendScript(buildCommand());

            TaskResponse response = shellCommandExecutor.run(shellActuatorBuilder, taskCallBack);
            setExitStatusCode(response.getExitStatusCode());
            setAppIds(String.join(TaskConstants.COMMA, getApplicationIds()));
            setProcessId(response.getProcessId());
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            log.error("The current Jupyter task has been interrupted", e);
            setExitStatusCode(TaskConstants.EXIT_CODE_FAILURE);
            throw new TaskException("The current Jupyter task has been interrupted", e);
        } catch (Exception e) {
            log.error("jupyter task execution failure", e);
            exitStatusCode = -1;
            throw new TaskException("Execute jupyter task failed", e);
        }
    }

    @Override
    public void submitApplication() throws TaskException {

    }

    @Override
    public void trackApplicationStatus() throws TaskException {

    }

    /**
     * command will be like: papermill [OPTIONS] NOTEBOOK_PATH [OUTPUT_PATH]
     */
    protected String buildCommand() throws IOException {

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Read the wrapped stack trace and the shell command output in worker logs for the root cause.
2. Verify jupyter/conda/pip are installed and on PATH for the worker user.
3. Run the logged jupyter command manually on the worker to reproduce the failure.
4. Check the notebook path and conda env/requirements referenced in task params exist.
5. If using packed/remote envs, verify the env directory and permissions.

Example fix

// before
} catch (Exception e) {
    throw new TaskException("Execute jupyter task failed", e);
}
// after
} catch (Exception e) {
    log.error("jupyter task failed, command: {}", String.join(" ", getJupyterTaskCommand()), e);
    throw new TaskException("Execute jupyter task failed: " + e.getMessage(), e);
}

// preflight: confirm binaries exist where the task will run
Process p = Runtime.getRuntime().exec(new String[]{"sh","-c","command -v jupyter && command -v conda"});
if (p.waitFor() != 0) throw new IllegalStateException("jupyter/conda not found on worker PATH");

static boolean isJupyterExecutionFailure(TaskException e) {
    return e.getMessage() != null && e.getMessage().startsWith("Execute jupyter task failed");
}

try {
    jupyterTask.handle(callBack);
} catch (TaskException e) {
    log.error("jupyter execution failed; root cause:", e.getCause());
    // retry only on transient env issues, fail fast on config issues
    if (isTransient(e.getCause())) retry();
}

When it happens

Trigger: shellCommandExecutor.run(jupyterTaskCommand) throws any Exception other than InterruptedException — command construction failure, jupyter/conda binary not found, process spawn error, or the TaskException thrown by handle's own init/prepare path propagating here.

Common situations: jupyter or conda not installed/on PATH on the worker, invalid notebook path at runtime, conda env activation failure, network issue downloading pip requirements, or malformed generated command line.

Related errors

• run java task error
• cancel application error
• Execute shell task error
• no master server available
• Backfill workflow failed: %s

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/ae80bee3b3c44580. Report an issue: GitHub.