{"record":{"id":"f66ec84f758b1f82","repo":"apache/flink","slug":"the-discovery-script-executed-for-over-d-ms","errorCode":null,"errorMessage":"The discovery script executed for over %d ms.","messagePattern":"The discovery script executed for over (.+?) ms\\.","errorType":"exception","errorClass":"TimeoutException","httpStatus":null,"severity":"error","filePath":"flink-external-resources/flink-external-resource-gpu/src/main/java/org/apache/flink/externalresource/gpu/GPUDriver.java","lineNumber":133,"sourceCode":"        return Collections.unmodifiableSet(gpuResources);\n    }\n\n    private String executeDiscoveryScript(File discoveryScript, long gpuAmount, String args)\n            throws Exception {\n        final String cmd = discoveryScript.getAbsolutePath() + \" \" + gpuAmount + \" \" + args;\n        final Process process = Runtime.getRuntime().exec(cmd);\n        try (final BufferedReader stdoutReader =\n                        new BufferedReader(\n                                new InputStreamReader(\n                                        process.getInputStream(), StandardCharsets.UTF_8));\n                final BufferedReader stderrReader =\n                        new BufferedReader(\n                                new InputStreamReader(\n                                        process.getErrorStream(), StandardCharsets.UTF_8))) {\n            final boolean hasProcessTerminated =\n                    process.waitFor(DISCOVERY_SCRIPT_TIMEOUT_MS, TimeUnit.MILLISECONDS);\n            if (!hasProcessTerminated) {\n                throw new TimeoutException(\n                        String.format(\n                                \"The discovery script executed for over %d ms.\",\n                                DISCOVERY_SCRIPT_TIMEOUT_MS));\n            }\n\n            final int exitVal = process.exitValue();\n            if (exitVal != 0) {\n                final String stdout =\n                        stdoutReader\n                                .lines()\n                                .collect(\n                                        StringBuilder::new,\n                                        StringBuilder::append,\n                                        StringBuilder::append)\n                                .toString();\n                final String stderr =\n                        stderrReader\n                                .lines()","sourceCodeStart":115,"sourceCodeEnd":151,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-external-resources/flink-external-resource-gpu/src/main/java/org/apache/flink/externalresource/gpu/GPUDriver.java#L115-L151","documentation":"Thrown as a TimeoutException by GPUDriver.retrieveResourceInfo when the discovery script process does not terminate within DISCOVERY_SCRIPT_TIMEOUT_MS milliseconds (a hard-coded constant in GPUDriver). The driver runs the script via Runtime.exec and waits with process.waitFor(timeout, MILLISECONDS); on expiry the process is abandoned and the error is raised.","triggerScenarios":"Calling retrieveResourceInfo (TaskManager startup requesting GPU resources) where the script blocks forever: nvidia-smi hanging on a broken GPU/driver, a script that prompts for input, a script waiting on a network resource, or GPU vendor tools deadlocking in containers without proper device access.","commonSituations":"nvidia-smi hangs when the NVIDIA driver version mismatches the kernel module; DCGM or nvidia-smi stuck inside a container missing /dev/nvidia* devices; a custom script that reads stdin or lacks a timeout around its own commands.","solutions":["Run the discovery script manually as the Flink user on an affected node and time it: 'time ./gpu-discovery.sh'","Fix the underlying hang: reinstall/match the NVIDIA driver, expose GPU devices to the container, or remove interactive prompts from the script","Add internal timeouts to the script so it always exits (e.g. 'timeout 10 nvidia-smi ...')","Verify the script works in the exact container image and cgroup the TaskManager runs in"],"exampleFix":"# before (inside gpu-discovery.sh)\nnvidia-smi --query-gpu=index --format=csv,noheader\n\n# after (always terminates)\ntimeout 10 nvidia-smi --query-gpu=index --format=csv,noheader || exit 1","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    Set<GPUInfo> gpus = driver.retrieveResourceInfo(amount);\n} catch (java.util.concurrent.TimeoutException e) {\n    // script hung; kill leftovers and retry once, then fail the task\n    log.warn(\"GPU discovery timed out, retrying\", e);\n}","preventionTips":["Wrap every command inside the discovery script with 'timeout'","Never let the script read stdin or prompt","Health-check nvidia-smi on GPU nodes so hangs are caught before job submission"],"tags":["gpu","timeout","process","external-resources"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}