apache/flink · error · FlinkRuntimeException

Could not find a public truncate method on the Hadoop File S

Error message

Could not find a public truncate method on the Hadoop File System.

What it means

On resume, Flink reflectively looks up FileSystem.truncate(Path, long) to truncate the temp file. If the method does not exist on the loaded Hadoop FileSystem class (NoSuchMethodException), this FlinkRuntimeException is thrown — the runtime Hadoop is too old (pre-2.7 API) or a Hadoop-compatible filesystem hides the method.

Source

Thrown at flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java:170

            truncated = truncate(fileSystem, path, recoverable.offset());
        } catch (Exception e) {
            throw new IOException("Problem while truncating file: " + path, e);
        }

        if (!truncated) {
            // Truncate did not complete immediately, we must wait for
            // the operation to complete and release the lease.
            revokeLeaseByFileSystem(fileSystem, path);
        }
    }

    private static void ensureTruncateInitialized() throws FlinkRuntimeException {
        if (HadoopUtils.isMinHadoopVersion(2, 7) && truncateHandle == null) {
            Method truncateMethod;
            try {
                truncateMethod = FileSystem.class.getMethod("truncate", Path.class, long.class);
            } catch (NoSuchMethodException e) {
                throw new FlinkRuntimeException(
                        "Could not find a public truncate method on the Hadoop File System.");
            }

            if (!Modifier.isPublic(truncateMethod.getModifiers())) {
                throw new FlinkRuntimeException(
                        "Could not find a public truncate method on the Hadoop File System.");
            }

            truncateHandle = truncateMethod;
        }
    }

    private static boolean truncate(final FileSystem hadoopFs, final Path file, final long length)
            throws IOException {
        if (!HadoopUtils.isMinHadoopVersion(2, 7)) {
            throw new IllegalStateException(
                    "Truncation is not available in hadoop version < 2.7 , You are on Hadoop "
                            + VersionInfo.getVersion());

View on GitHub (pinned to 2f3c205e92)

Solutions

Upgrade the Hadoop jars/classpath to 2.7+ (preferably 2.8+/3.x) so FileSystem.truncate exists
Remove conflicting old hadoop-common jars from user fat-jars so Flink loads the runtime cluster's Hadoop
If the filesystem cannot support truncate, avoid the HDFS recoverable resume path (use a non-resuming sink or a filesystem with truncate support)

Defensive patterns

Strategy: validation

Validate before calling

// guard before using recoverable HDFS output
if (!HadoopUtils.isMinHadoopVersion(2, 7)) {
    throw new UnsupportedOperationException("HDFS resume requires Hadoop >= 2.7");
}
try {
    org.apache.hadoop.fs.FileSystem.class.getMethod("truncate", org.apache.hadoop.fs.Path.class, long.class);
} catch (NoSuchMethodException e) {
    throw new UnsupportedOperationException("Runtime Hadoop lacks FileSystem.truncate", e);
}

Prevention

Standardize on Hadoop 2.7+ client jars
Fail fast at job startup with an explicit Hadoop version check
Audit for stray old hadoop-common jars in user dependencies

When it happens

Trigger: ensureTruncateInitialized() running FileSystem.class.getMethod("truncate", Path.class, long.class) and catching NoSuchMethodException: Hadoop < 2.7 on the classpath, or a third-party FileSystem class without the truncate API.

Common situations: Ancient Hadoop 2.x/1.x jars leaked into lib/; custom Hadoop-compatible filesystem shims that did not implement truncate; dependency downgrades in user fat-jars shading an old hadoop-common.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/ad21c22e3fd7831e. Report an issue: GitHub.