{"record":{"id":"6103274cc5bd236c","repo":"apache/flink","slug":"truncate-failed-tempfile-requested-recovera","errorCode":null,"errorMessage":"Truncate failed: ${tempFile} (requested=${recoverable.offset()} ,size=${pos})","messagePattern":"Truncate failed: (.+?) \\(requested=(.+?) ,size=(.+?)\\)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java","lineNumber":119,"sourceCode":"\n    HadoopRecoverableFsDataOutputStream(FileSystem fs, HadoopFsRecoverable recoverable)\n            throws IOException {\n\n        ensureTruncateInitialized();\n\n        this.fs = checkNotNull(fs);\n        this.targetFile = checkNotNull(recoverable.targetFile());\n        this.tempFile = checkNotNull(recoverable.tempFile());\n\n        safelyTruncateFile(fs, tempFile, recoverable);\n\n        out = fs.append(tempFile);\n\n        // sanity check\n        long pos = out.getPos();\n        if (pos != recoverable.offset()) {\n            IOUtils.closeQuietly(out);\n            throw new IOException(\n                    \"Truncate failed: \"\n                            + tempFile\n                            + \" (requested=\"\n                            + recoverable.offset()\n                            + \" ,size=\"\n                            + pos\n                            + ')');\n        }\n    }\n\n    @Override\n    protected Committer createCommitterFromResumeRecoverable(HadoopFsRecoverable recoverable) {\n        return new HadoopFsCommitter(fs, recoverable);\n    }\n\n    // ------------------------------------------------------------------------\n    //  Reflection utils for truncation\n    //    These are needed to compile against Hadoop versions before","sourceCodeStart":101,"sourceCodeEnd":137,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L101-L137","documentation":"When resuming a Hadoop recoverable output stream, the file is truncated to the recoverable offset and reopened for append; the code then sanity-checks that the append stream's position equals the requested offset. A mismatch means HDFS truncate had not fully completed (it is asynchronous) or the temp file is not in the expected state, so the stream refuses to continue from a wrong position.","triggerScenarios":"Constructing HadoopRecoverableFsDataOutputStream / resuming from a HadoopFsRecoverable where fs.append(tempFile).getPos() != recoverable.offset() — typically because HDFS truncate was still in progress (truncated=false path) when append opened the file.","commonSituations":"Task failover resuming a file sink on HDFS immediately after truncate; HDFS-3107-style asynchronous truncate semantics where a concurrent reader sees the old length; state recorded after a partial flush so offset disagrees with actual file size.","solutions":["Retry the resume: on failover, wait/retry until HDFS reports the truncated length before appending (truncate completion is async in HDFS)","Ensure HDFS client and cluster run a version where truncate+append interplay is reliable (2.7+; prefer 2.8/3.x)","If the temp file is corrupted beyond repair, discard the in-flight part and rewrite from the last successful checkpoint"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before resuming, confirm the file already has the expected length\nlong expected = recoverable.offset();\nlong actual = fs.getFileStatus(tempFile).getLen();\nif (actual != expected) {\n    // wait or truncate explicitly until lengths match before constructing the stream\n}","typeGuard":null,"tryCatchPattern":"try {\n    out = writer.recover(recoverable);\n} catch (IOException e) {\n    if (e.getMessage().contains(\"Truncate failed\")) {\n        // async HDFS truncate not settled: poll file length, then retry recover once\n    }\n}","preventionTips":["On failover, poll until HDFS reports the truncated length before resuming","Use HDFS 2.8+/3.x for truncate+append reliability","Keep regular checkpoints so a bad temp file can be rewritten instead of resumed"],"tags":["hdfs","truncate","recovery","file-sink"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}