apache/flink · error · TimeoutException

The discovery script executed for over %d ms.

Error message

The discovery script executed for over %d ms.

What it means

Thrown as a TimeoutException by GPUDriver.retrieveResourceInfo when the discovery script process does not terminate within DISCOVERY_SCRIPT_TIMEOUT_MS milliseconds (a hard-coded constant in GPUDriver). The driver runs the script via Runtime.exec and waits with process.waitFor(timeout, MILLISECONDS); on expiry the process is abandoned and the error is raised.

Source

Thrown at flink-external-resources/flink-external-resource-gpu/src/main/java/org/apache/flink/externalresource/gpu/GPUDriver.java:133

        return Collections.unmodifiableSet(gpuResources);
    }

    private String executeDiscoveryScript(File discoveryScript, long gpuAmount, String args)
            throws Exception {
        final String cmd = discoveryScript.getAbsolutePath() + " " + gpuAmount + " " + args;
        final Process process = Runtime.getRuntime().exec(cmd);
        try (final BufferedReader stdoutReader =
                        new BufferedReader(
                                new InputStreamReader(
                                        process.getInputStream(), StandardCharsets.UTF_8));
                final BufferedReader stderrReader =
                        new BufferedReader(
                                new InputStreamReader(
                                        process.getErrorStream(), StandardCharsets.UTF_8))) {
            final boolean hasProcessTerminated =
                    process.waitFor(DISCOVERY_SCRIPT_TIMEOUT_MS, TimeUnit.MILLISECONDS);
            if (!hasProcessTerminated) {
                throw new TimeoutException(
                        String.format(
                                "The discovery script executed for over %d ms.",
                                DISCOVERY_SCRIPT_TIMEOUT_MS));
            }

            final int exitVal = process.exitValue();
            if (exitVal != 0) {
                final String stdout =
                        stdoutReader
                                .lines()
                                .collect(
                                        StringBuilder::new,
                                        StringBuilder::append,
                                        StringBuilder::append)
                                .toString();
                final String stderr =
                        stderrReader
                                .lines()

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Run the discovery script manually as the Flink user on an affected node and time it: 'time ./gpu-discovery.sh'
2. Fix the underlying hang: reinstall/match the NVIDIA driver, expose GPU devices to the container, or remove interactive prompts from the script
3. Add internal timeouts to the script so it always exits (e.g. 'timeout 10 nvidia-smi ...')
4. Verify the script works in the exact container image and cgroup the TaskManager runs in

Example fix

# before (inside gpu-discovery.sh)
nvidia-smi --query-gpu=index --format=csv,noheader

# after (always terminates)
timeout 10 nvidia-smi --query-gpu=index --format=csv,noheader || exit 1

try {
    Set<GPUInfo> gpus = driver.retrieveResourceInfo(amount);
} catch (java.util.concurrent.TimeoutException e) {
    // script hung; kill leftovers and retry once, then fail the task
    log.warn("GPU discovery timed out, retrying", e);
}

When it happens

Trigger: Calling retrieveResourceInfo (TaskManager startup requesting GPU resources) where the script blocks forever: nvidia-smi hanging on a broken GPU/driver, a script that prompts for input, a script waiting on a network resource, or GPU vendor tools deadlocking in containers without proper device access.

Common situations: nvidia-smi hangs when the NVIDIA driver version mismatches the kernel module; DCGM or nvidia-smi stuck inside a container missing /dev/nvidia* devices; a custom script that reads stdin or lacks a timeout around its own commands.

Related errors

• The gpu discovery script does not exist in path %s.
• The discovery script %s is not executable.
• Discovery script exit with non-zero return code: %s.
• Do not support external resource in current environment
• The timeout for opening the input splits must be positive or

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/f66ec84f758b1f82. Report an issue: GitHub.