{"record":{"id":"e6a859393032c42f","repo":"apache/flink","slug":"truncation-of-file-failed-because-of-access-linkin","errorCode":null,"errorMessage":"Truncation of file failed because of access/linking problems with Hadoop's truncate call. This is most likely a dependency conflict or class loading problem.","messagePattern":"Truncation of file failed because of access/linking problems with Hadoop's truncate call\\. This is most likely a dependency conflict or class loading problem\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java","lineNumber":197,"sourceCode":"            truncateHandle = truncateMethod;\n        }\n    }\n\n    private static boolean truncate(final FileSystem hadoopFs, final Path file, final long length)\n            throws IOException {\n        if (!HadoopUtils.isMinHadoopVersion(2, 7)) {\n            throw new IllegalStateException(\n                    \"Truncation is not available in hadoop version < 2.7 , You are on Hadoop \"\n                            + VersionInfo.getVersion());\n        }\n\n        if (truncateHandle != null) {\n            try {\n                return (Boolean) truncateHandle.invoke(hadoopFs, file, length);\n            } catch (InvocationTargetException e) {\n                ExceptionUtils.rethrowIOException(e.getTargetException());\n            } catch (Throwable t) {\n                throw new IOException(\n                        \"Truncation of file failed because of access/linking problems with Hadoop's truncate call. \"\n                                + \"This is most likely a dependency conflict or class loading problem.\");\n            }\n        } else {\n            throw new IllegalStateException(\"Truncation handle has not been initialized\");\n        }\n        return false;\n    }\n\n    // ------------------------------------------------------------------------\n    //  Committer\n    // ------------------------------------------------------------------------\n\n    /**\n     * Implementation of a committer for the Hadoop File System abstraction. This implementation\n     * commits by renaming the temp file to the final file path. The temp file is truncated before\n     * renaming in case there is trailing garbage data.\n     */","sourceCodeStart":179,"sourceCodeEnd":215,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L179-L215","documentation":"The cached Method handle for Hadoop truncate exists, but invoking it throws a non-InvocationTargetException Throwable (e.g. IllegalAccessException or LinkageError-like failures). Flink wraps this as an IOException stating it is most likely a dependency conflict or class loading problem — the class that declared truncate is not linkable with the class actually loaded at runtime.","triggerScenarios":"truncateHandle.invoke(hadoopFs, file, length) throwing IllegalAccessException/LinkageError: mixed Hadoop versions where FileSystem was resolved from one jar and the runtime instance from another, or module/access restrictions on the JDK.","commonSituations":"Multiple hadoop-common versions across lib/ and user jars; shaded Hadoop with inconsistent relocation; running on newer JDKs where reflective access to JDK-internal or sealed packages is denied.","solutions":["Unify Hadoop dependencies: exactly one hadoop-common version across Flink lib/ and the job jar (check with dependency:tree)","Prefer Flink's bundled/prebuilt Hadoop filesystem plugins over user-bundled Hadoop","If on a restrictive JDK, ensure reflective access is permitted (no illegitimate module locks for org.apache.hadoop classes)","Verify with ClassLoader inspection that FileSystem and the instance come from the same jar"],"exampleFix":"# remove the conflicting old jar from the user fat-jar and rely on lib/\nmvn dependency:tree | grep hadoop-common\n# keep exactly one version","handlingStrategy":"try-catch","validationCode":"// verify the FileSystem instance and class come from the same classloader/jar\nClass<?> fsClass = fs.getClass();\nClass<?> declClass = org.apache.hadoop.fs.FileSystem.class;\nif (fsClass.getClassLoader() != declClass.getClassLoader()) {\n    // mixed Hadoop jars on classpath: unify before using truncate-based resume\n}","typeGuard":null,"tryCatchPattern":"try {\n    out = writer.recover(recoverable);\n} catch (IOException e) {\n    if (e.getMessage().contains(\"dependency conflict or class loading problem\")) {\n        // classpath issue: dedupe hadoop jars, redeploy; not retryable in-process\n    }\n}","preventionTips":["Mark all hadoop-* dependencies provided/compile-only in job jars","Run one Hadoop version cluster-wide and verify with dependency:tree","Add CI checks that fail builds bundling duplicate hadoop-common classes"],"tags":["hdfs","dependency-conflict","classloading","truncate"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}