apache/flink · critical · RuntimeException

Exception encountered during finding the flink-python jar. T

Error message

Exception encountered during finding the flink-python jar. This should not happen.

What it means

Thrown when an IOException occurs while walking the Flink 'opt' directory to locate a jar (typically the flink-python jar). The method getOptJar uses Files.walkFileTree on the FLINK_OPT_DIR environment variable path. If that directory walk fails (directory missing, unreadable, or IO error), this RuntimeException is thrown.

Source

Thrown at flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgramUtils.java:268

    private static URL getOptJar(String jarName) {
        String flinkOptPath = System.getenv(ConfigConstants.ENV_FLINK_OPT_DIR);
        final List<Path> optJarPath = new ArrayList<>();
        try {
            Files.walkFileTree(
                    FileSystems.getDefault().getPath(flinkOptPath),
                    new SimpleFileVisitor<Path>() {
                        @Override
                        public FileVisitResult visitFile(Path file, BasicFileAttributes attrs)
                                throws IOException {
                            FileVisitResult result = super.visitFile(file, attrs);
                            if (file.getFileName().toString().startsWith(jarName)) {
                                optJarPath.add(file);
                            }
                            return result;
                        }
                    });
        } catch (IOException e) {
            throw new RuntimeException(
                    "Exception encountered during finding the flink-python jar. This should not happen.",
                    e);
        }

        if (optJarPath.size() != 1) {
            throw new RuntimeException(
                    String.format("Found " + optJarPath.size() + " %s jar.", jarName));
        }

        try {
            return optJarPath.get(0).toUri().toURL();
        } catch (MalformedURLException e) {
            throw new RuntimeException("URL is invalid. This should not happen.", e);
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

Verify FLINK_HOME points to a complete Flink installation.
Check that $FLINK_HOME/opt exists and is readable.
Ensure FLINK_OPT_DIR env var (if set) points to the correct opt directory.
Reinstall Flink from a complete distribution if opt/ is missing.

Example fix

# before: FLINK_HOME points to incomplete install
export FLINK_HOME=/opt/flink-incomplete

# after: point to full installation
export FLINK_HOME=/opt/flink
ls $FLINK_HOME/opt/  # should contain flink-python jars

Defensive patterns

Strategy: validation

Validate before calling

String optDir = System.getenv(ConfigConstants.ENV_FLINK_OPT_DIR);
if (optDir == null || !new File(optDir).isDirectory()) {
    throw new IllegalStateException("FLINK_OPT_DIR is not set or invalid: " + optDir);
}

Try / catch

try {
    // code that triggers PyFlink / getOptJar
} catch (RuntimeException e) {
    if (e.getMessage().contains("finding the flink-python jar")) {
        // check FLINK_HOME and opt/ directory
    }
    throw e;
}

Prevention

Set FLINK_HOME to a complete Flink installation.
Verify $FLINK_HOME/opt is present and readable.
Do not set FLINK_OPT_DIR to a custom path unless it mirrors the standard opt/ layout.

When it happens

Trigger: The FLINK_OPT_DIR environment variable points to a path that does not exist, is not a directory, or has an I/O error during traversal. Triggered when PyFlink or the SQL client tries to locate the python jar at startup.

Common situations: FLINK_HOME or FLINK_OPT_DIR set incorrectly, partial Flink installation missing the opt/ directory, or filesystem permissions preventing directory listing.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/9f7ea3d9ceb8441a. Report an issue: GitHub.