apache/flink · error · IOException

Truncate failed: ${tempFile} (requested=${recoverable.offset

Error message

Truncate failed: ${tempFile} (requested=${recoverable.offset()} ,size=${pos})

What it means

When resuming a Hadoop recoverable output stream, the file is truncated to the recoverable offset and reopened for append; the code then sanity-checks that the append stream's position equals the requested offset. A mismatch means HDFS truncate had not fully completed (it is asynchronous) or the temp file is not in the expected state, so the stream refuses to continue from a wrong position.

Source

Thrown at flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java:119

    HadoopRecoverableFsDataOutputStream(FileSystem fs, HadoopFsRecoverable recoverable)
            throws IOException {

        ensureTruncateInitialized();

        this.fs = checkNotNull(fs);
        this.targetFile = checkNotNull(recoverable.targetFile());
        this.tempFile = checkNotNull(recoverable.tempFile());

        safelyTruncateFile(fs, tempFile, recoverable);

        out = fs.append(tempFile);

        // sanity check
        long pos = out.getPos();
        if (pos != recoverable.offset()) {
            IOUtils.closeQuietly(out);
            throw new IOException(
                    "Truncate failed: "
                            + tempFile
                            + " (requested="
                            + recoverable.offset()
                            + " ,size="
                            + pos
                            + ')');
        }
    }

    @Override
    protected Committer createCommitterFromResumeRecoverable(HadoopFsRecoverable recoverable) {
        return new HadoopFsCommitter(fs, recoverable);
    }

    // ------------------------------------------------------------------------
    //  Reflection utils for truncation
    //    These are needed to compile against Hadoop versions before

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Retry the resume: on failover, wait/retry until HDFS reports the truncated length before appending (truncate completion is async in HDFS)
2. Ensure HDFS client and cluster run a version where truncate+append interplay is reliable (2.7+; prefer 2.8/3.x)
3. If the temp file is corrupted beyond repair, discard the in-flight part and rewrite from the last successful checkpoint

// before resuming, confirm the file already has the expected length
long expected = recoverable.offset();
long actual = fs.getFileStatus(tempFile).getLen();
if (actual != expected) {
    // wait or truncate explicitly until lengths match before constructing the stream
}

try {
    out = writer.recover(recoverable);
} catch (IOException e) {
    if (e.getMessage().contains("Truncate failed")) {
        // async HDFS truncate not settled: poll file length, then retry recover once
    }
}

When it happens

Trigger: Constructing HadoopRecoverableFsDataOutputStream / resuming from a HadoopFsRecoverable where fs.append(tempFile).getPos() != recoverable.offset() — typically because HDFS truncate was still in progress (truncated=false path) when append opened the file.

Common situations: Task failover resuming a file sink on HDFS immediately after truncate; HDFS-3107-style asynchronous truncate semantics where a concurrent reader sees the old length; state recorded after a partial flush so offset disagrees with actual file size.

Related errors

• Problem while truncating file: ${path}
• Cannot clean commit: Staging file does not exist.
• Cannot clean commit: File has trailing junk data.
• can not recover from the pendingFileRecoverable
• Unable to recover the job as the expected {} file is not fou

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/6103274cc5bd236c. Report an issue: GitHub.