{"record":{"id":"ad21c22e3fd7831e","repo":"apache/flink","slug":"could-not-find-a-public-truncate-method-on-the-had","errorCode":null,"errorMessage":"Could not find a public truncate method on the Hadoop File System.","messagePattern":"Could not find a public truncate method on the Hadoop File System\\.","errorType":"exception","errorClass":"FlinkRuntimeException","httpStatus":null,"severity":"error","filePath":"flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java","lineNumber":170,"sourceCode":"            truncated = truncate(fileSystem, path, recoverable.offset());\n        } catch (Exception e) {\n            throw new IOException(\"Problem while truncating file: \" + path, e);\n        }\n\n        if (!truncated) {\n            // Truncate did not complete immediately, we must wait for\n            // the operation to complete and release the lease.\n            revokeLeaseByFileSystem(fileSystem, path);\n        }\n    }\n\n    private static void ensureTruncateInitialized() throws FlinkRuntimeException {\n        if (HadoopUtils.isMinHadoopVersion(2, 7) && truncateHandle == null) {\n            Method truncateMethod;\n            try {\n                truncateMethod = FileSystem.class.getMethod(\"truncate\", Path.class, long.class);\n            } catch (NoSuchMethodException e) {\n                throw new FlinkRuntimeException(\n                        \"Could not find a public truncate method on the Hadoop File System.\");\n            }\n\n            if (!Modifier.isPublic(truncateMethod.getModifiers())) {\n                throw new FlinkRuntimeException(\n                        \"Could not find a public truncate method on the Hadoop File System.\");\n            }\n\n            truncateHandle = truncateMethod;\n        }\n    }\n\n    private static boolean truncate(final FileSystem hadoopFs, final Path file, final long length)\n            throws IOException {\n        if (!HadoopUtils.isMinHadoopVersion(2, 7)) {\n            throw new IllegalStateException(\n                    \"Truncation is not available in hadoop version < 2.7 , You are on Hadoop \"\n                            + VersionInfo.getVersion());","sourceCodeStart":152,"sourceCodeEnd":188,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L152-L188","documentation":"On resume, Flink reflectively looks up FileSystem.truncate(Path, long) to truncate the temp file. If the method does not exist on the loaded Hadoop FileSystem class (NoSuchMethodException), this FlinkRuntimeException is thrown — the runtime Hadoop is too old (pre-2.7 API) or a Hadoop-compatible filesystem hides the method.","triggerScenarios":"ensureTruncateInitialized() running FileSystem.class.getMethod(\"truncate\", Path.class, long.class) and catching NoSuchMethodException: Hadoop < 2.7 on the classpath, or a third-party FileSystem class without the truncate API.","commonSituations":"Ancient Hadoop 2.x/1.x jars leaked into lib/; custom Hadoop-compatible filesystem shims that did not implement truncate; dependency downgrades in user fat-jars shading an old hadoop-common.","solutions":["Upgrade the Hadoop jars/classpath to 2.7+ (preferably 2.8+/3.x) so FileSystem.truncate exists","Remove conflicting old hadoop-common jars from user fat-jars so Flink loads the runtime cluster's Hadoop","If the filesystem cannot support truncate, avoid the HDFS recoverable resume path (use a non-resuming sink or a filesystem with truncate support)"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// guard before using recoverable HDFS output\nif (!HadoopUtils.isMinHadoopVersion(2, 7)) {\n    throw new UnsupportedOperationException(\"HDFS resume requires Hadoop >= 2.7\");\n}\ntry {\n    org.apache.hadoop.fs.FileSystem.class.getMethod(\"truncate\", org.apache.hadoop.fs.Path.class, long.class);\n} catch (NoSuchMethodException e) {\n    throw new UnsupportedOperationException(\"Runtime Hadoop lacks FileSystem.truncate\", e);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Standardize on Hadoop 2.7+ client jars","Fail fast at job startup with an explicit Hadoop version check","Audit for stray old hadoop-common jars in user dependencies"],"tags":["hdfs","hadoop-version","reflection","truncate"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}