apache/flink · error · FlinkException

Discovery script exit with non-zero return code: %s.

Error message

Discovery script exit with non-zero return code: %s.

What it means

Thrown as a FlinkException by GPUDriver.retrieveResourceInfo when the discovery script terminates with a non-zero exit code. The driver captures the script's STDOUT and STDERR and logs them at WARN level ('Discovery script exit with {exitVal}. STDOUT/STDERR') immediately before throwing, so the root cause is in the log, not in the exception message.

Source

Thrown at flink-external-resources/flink-external-resource-gpu/src/main/java/org/apache/flink/externalresource/gpu/GPUDriver.java:162

                                .collect(
                                        StringBuilder::new,
                                        StringBuilder::append,
                                        StringBuilder::append)
                                .toString();
                final String stderr =
                        stderrReader
                                .lines()
                                .collect(
                                        StringBuilder::new,
                                        StringBuilder::append,
                                        StringBuilder::append)
                                .toString();
                LOG.warn(
                        "Discovery script exit with {}.\nSTDOUT: {}\nSTDERR: {}",
                        exitVal,
                        stdout,
                        stderr);
                throw new FlinkException(
                        String.format(
                                "Discovery script exit with non-zero return code: %s.", exitVal));
            }
            Object[] stdout = stdoutReader.lines().toArray();
            if (stdout.length > 1) {
                LOG.warn(
                        "The output of the discovery script should only contain one single line. Finding {} lines with content: {}. Will only keep the first line.",
                        stdout.length,
                        Arrays.toString(stdout));
            }
            if (stdout.length == 0) {
                return "";
            }
            return (String) stdout[0];
        } finally {
            process.destroyForcibly();
        }
    }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Read the WARN log line right before this exception to see the script's exit code, STDOUT and STDERR
2. Reproduce by running the script manually as the Flink user: './gpu-discovery.sh; echo $?'
3. Fix the failing command inside the script (driver install, device passthrough, missing tools)
4. Make the script exit 0 with empty output on nodes where a zero-GPU result is legitimate, if that matches your setup

try {
    driver.retrieveResourceInfo(amount);
} catch (org.apache.flink.util.FlinkException e) {
    // STDOUT/STDERR were logged at WARN just before this; inspect logs for root cause
    log.error("GPU discovery script failed; see preceding WARN with STDOUT/STDERR", e);
}

When it happens

Trigger: Calling retrieveResourceInfo where the script itself fails: nvidia-smi returning non-zero (no GPUs visible, driver error), a shell script with 'set -e' hitting a failed command, missing interpreter (bad shebang path), or a Python-based script raising an exception.

Common situations: GPU node drained or devices not passed to the container (nvidia-smi exits 6 'No GPUs were found'); shebang pointing to a python not present in the image; script assumes tools (jq, curl) that are missing on workers.

Related errors

• The gpu discovery script does not exist in path %s.
• The discovery script %s is not executable.
• The discovery script executed for over %d ms.
• Do not support external resource in current environment
• Bulk copy failed

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/d64ed6cd46729bd9. Report an issue: GitHub.