apache/flink · error · IOException

Error recovering writer: Downloading the last data chunk fil

Error message

Error recovering writer: Downloading the last data chunk file gives incorrect length. File=%d bytes, Stream=%d bytes

What it means

HadoopS3AccessHelper.copyLastPartToTargetAndSeek downloads the trailing data chunk of an interrupted multipart upload during writer recovery, then sanity-checks that the number of bytes read equals the recorded local temp-file length. A mismatch means the S3 object's last part no longer matches the local metadata recorded in the recoverable state, so appending would corrupt the output.

Source

Thrown at flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/HadoopS3AccessHelper.java:130

    @Override
    public long getObject(String key, File targetLocation) throws IOException {
        long numBytes = 0L;
        try (final OutputStream outStream = new FileOutputStream(targetLocation);
                final org.apache.hadoop.fs.FSDataInputStream inStream =
                        s3a.open(new org.apache.hadoop.fs.Path('/' + key))) {
            final byte[] buffer = new byte[32 * 1024];

            int numRead;
            while ((numRead = inStream.read(buffer)) != -1) {
                outStream.write(buffer, 0, numRead);
                numBytes += numRead;
            }
        }

        // some sanity checks
        if (numBytes != targetLocation.length()) {
            throw new IOException(
                    String.format(
                            "Error recovering writer: "
                                    + "Downloading the last data chunk file gives incorrect length. "
                                    + "File=%d bytes, Stream=%d bytes",
                            targetLocation.length(), numBytes));
        }

        return numBytes;
    }

    @Override
    public ObjectMetadata getObjectMetadata(String key) throws IOException {
        try {
            return s3a.getObjectMetadata(new Path('/' + key));
        } catch (SdkBaseException e) {
            throw S3AUtils.translateException("getObjectMetadata", key, e);
        }
    }

View on GitHub (pinned to 2f3c205e92)

Solutions

Ensure only one writer at a time works on the same output key and the same local tmp directory; concurrent resumes corrupt length bookkeeping.
Verify the local temp file referenced by the recoverable still exists with its original length and was not touched by cleanup jobs.
Check S3 bucket lifecycle rules or external tools are not mutating in-progress multipart upload parts.
If the state is irrecoverable, abandon the resume, delete the partial upload (AbortMultipartUpload), and rewrite the file from scratch.

Defensive patterns

Strategy: validation

Validate before calling

// before attempting resume, verify local temp file still matches recorded length
File tmp = new File(recoverable.tempFile());
if (!tmp.exists() || tmp.length() != recoverable.offset()) {
    // refuse to resume; the sanity check in HadoopS3AccessHelper would fail
    throw new IOException("Local temp file drift detected; refusing resume");
}

Try / catch

try {
    writer.recover(recoverable);
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("incorrect length")) {
        // state mismatch: abort the multipart upload and rewrite from scratch
    } else {
        throw e;
    }
}

Prevention

Ensure a single writer per output key; never resume the same recoverable from two tasks.
Protect local tmp directories from cleanup agents and disk-full conditions.
Prevent external processes from mutating in-progress multipart upload parts.

When it happens

Trigger: Recovering an S3RecoverableWriter for resume when the last uploaded part in S3 has a different size than targetLocation.length() recorded at snapshot time — e.g. the object was modified, overwritten, lifecycle-trimmed, or the temp file metadata is stale.

Common situations: Multipart upload resumed after the underlying S3 object or its multipart upload parts were altered by another process; the local temp file referenced by the recoverable was truncated or recreated; clock/part-numbering drift after an aborted upload retry; or concurrent writers targeting the same key.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/40b2878c0483480e. Report an issue: GitHub.