{"record":{"id":"d64ed6cd46729bd9","repo":"apache/flink","slug":"discovery-script-exit-with-non-zero-return-code","errorCode":null,"errorMessage":"Discovery script exit with non-zero return code: %s.","messagePattern":"Discovery script exit with non-zero return code: (.+?)\\.","errorType":"exception","errorClass":"FlinkException","httpStatus":null,"severity":"error","filePath":"flink-external-resources/flink-external-resource-gpu/src/main/java/org/apache/flink/externalresource/gpu/GPUDriver.java","lineNumber":162,"sourceCode":"                                .collect(\n                                        StringBuilder::new,\n                                        StringBuilder::append,\n                                        StringBuilder::append)\n                                .toString();\n                final String stderr =\n                        stderrReader\n                                .lines()\n                                .collect(\n                                        StringBuilder::new,\n                                        StringBuilder::append,\n                                        StringBuilder::append)\n                                .toString();\n                LOG.warn(\n                        \"Discovery script exit with {}.\\nSTDOUT: {}\\nSTDERR: {}\",\n                        exitVal,\n                        stdout,\n                        stderr);\n                throw new FlinkException(\n                        String.format(\n                                \"Discovery script exit with non-zero return code: %s.\", exitVal));\n            }\n            Object[] stdout = stdoutReader.lines().toArray();\n            if (stdout.length > 1) {\n                LOG.warn(\n                        \"The output of the discovery script should only contain one single line. Finding {} lines with content: {}. Will only keep the first line.\",\n                        stdout.length,\n                        Arrays.toString(stdout));\n            }\n            if (stdout.length == 0) {\n                return \"\";\n            }\n            return (String) stdout[0];\n        } finally {\n            process.destroyForcibly();\n        }\n    }","sourceCodeStart":144,"sourceCodeEnd":180,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-external-resources/flink-external-resource-gpu/src/main/java/org/apache/flink/externalresource/gpu/GPUDriver.java#L144-L180","documentation":"Thrown as a FlinkException by GPUDriver.retrieveResourceInfo when the discovery script terminates with a non-zero exit code. The driver captures the script's STDOUT and STDERR and logs them at WARN level ('Discovery script exit with {exitVal}. STDOUT/STDERR') immediately before throwing, so the root cause is in the log, not in the exception message.","triggerScenarios":"Calling retrieveResourceInfo where the script itself fails: nvidia-smi returning non-zero (no GPUs visible, driver error), a shell script with 'set -e' hitting a failed command, missing interpreter (bad shebang path), or a Python-based script raising an exception.","commonSituations":"GPU node drained or devices not passed to the container (nvidia-smi exits 6 'No GPUs were found'); shebang pointing to a python not present in the image; script assumes tools (jq, curl) that are missing on workers.","solutions":["Read the WARN log line right before this exception to see the script's exit code, STDOUT and STDERR","Reproduce by running the script manually as the Flink user: './gpu-discovery.sh; echo $?'","Fix the failing command inside the script (driver install, device passthrough, missing tools)","Make the script exit 0 with empty output on nodes where a zero-GPU result is legitimate, if that matches your setup"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    driver.retrieveResourceInfo(amount);\n} catch (org.apache.flink.util.FlinkException e) {\n    // STDOUT/STDERR were logged at WARN just before this; inspect logs for root cause\n    log.error(\"GPU discovery script failed; see preceding WARN with STDOUT/STDERR\", e);\n}","preventionTips":["Test the discovery script in CI on an image identical to production","Pin interpreter paths in the shebang to versions present in the image","Log script stderr yourself during development to shorten diagnosis"],"tags":["gpu","process-exit-code","external-resources","diagnostics"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}