apache/flink · error · IOException

Truncation of file failed because of access/linking problems

Error message

Truncation of file failed because of access/linking problems with Hadoop's truncate call. This is most likely a dependency conflict or class loading problem.

What it means

The cached Method handle for Hadoop truncate exists, but invoking it throws a non-InvocationTargetException Throwable (e.g. IllegalAccessException or LinkageError-like failures). Flink wraps this as an IOException stating it is most likely a dependency conflict or class loading problem — the class that declared truncate is not linkable with the class actually loaded at runtime.

Source

Thrown at flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java:197

            truncateHandle = truncateMethod;
        }
    }

    private static boolean truncate(final FileSystem hadoopFs, final Path file, final long length)
            throws IOException {
        if (!HadoopUtils.isMinHadoopVersion(2, 7)) {
            throw new IllegalStateException(
                    "Truncation is not available in hadoop version < 2.7 , You are on Hadoop "
                            + VersionInfo.getVersion());
        }

        if (truncateHandle != null) {
            try {
                return (Boolean) truncateHandle.invoke(hadoopFs, file, length);
            } catch (InvocationTargetException e) {
                ExceptionUtils.rethrowIOException(e.getTargetException());
            } catch (Throwable t) {
                throw new IOException(
                        "Truncation of file failed because of access/linking problems with Hadoop's truncate call. "
                                + "This is most likely a dependency conflict or class loading problem.");
            }
        } else {
            throw new IllegalStateException("Truncation handle has not been initialized");
        }
        return false;
    }

    // ------------------------------------------------------------------------
    //  Committer
    // ------------------------------------------------------------------------

    /**
     * Implementation of a committer for the Hadoop File System abstraction. This implementation
     * commits by renaming the temp file to the final file path. The temp file is truncated before
     * renaming in case there is trailing garbage data.
     */

View on GitHub (pinned to 2f3c205e92)

Solutions

Unify Hadoop dependencies: exactly one hadoop-common version across Flink lib/ and the job jar (check with dependency:tree)
Prefer Flink's bundled/prebuilt Hadoop filesystem plugins over user-bundled Hadoop
If on a restrictive JDK, ensure reflective access is permitted (no illegitimate module locks for org.apache.hadoop classes)
Verify with ClassLoader inspection that FileSystem and the instance come from the same jar

Example fix

# remove the conflicting old jar from the user fat-jar and rely on lib/
mvn dependency:tree | grep hadoop-common
# keep exactly one version

Defensive patterns

Strategy: try-catch

Validate before calling

// verify the FileSystem instance and class come from the same classloader/jar
Class<?> fsClass = fs.getClass();
Class<?> declClass = org.apache.hadoop.fs.FileSystem.class;
if (fsClass.getClassLoader() != declClass.getClassLoader()) {
    // mixed Hadoop jars on classpath: unify before using truncate-based resume
}

Try / catch

try {
    out = writer.recover(recoverable);
} catch (IOException e) {
    if (e.getMessage().contains("dependency conflict or class loading problem")) {
        // classpath issue: dedupe hadoop jars, redeploy; not retryable in-process
    }
}

Prevention

Mark all hadoop-* dependencies provided/compile-only in job jars
Run one Hadoop version cluster-wide and verify with dependency:tree
Add CI checks that fail builds bundling duplicate hadoop-common classes

When it happens

Trigger: truncateHandle.invoke(hadoopFs, file, length) throwing IllegalAccessException/LinkageError: mixed Hadoop versions where FileSystem was resolved from one jar and the runtime instance from another, or module/access restrictions on the JDK.

Common situations: Multiple hadoop-common versions across lib/ and user jars; shaded Hadoop with inconsistent relocation; running on newer JDKs where reflective access to JDK-internal or sealed packages is denied.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/e6a859393032c42f. Report an issue: GitHub.