apache/flink · error · IOException

Error recovering writer: Downloading the last data chunk fil

Error message

Error recovering writer: Downloading the last data chunk file gives incorrect length.File length is %d bytes, RecoveryData indicates %d bytes

What it means

Error "Error recovering writer: Downloading the last data chunk file gives incorrect length.File length is %d bytes, RecoveryData indicates %d bytes" thrown in apache/flink.

Source

Thrown at flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/writer/S3RecoverableMultipartUploadFactory.java:99

                recoverable.parts(),
                recoverable.numBytesInParts(),
                incompletePart);
    }

    private Optional<File> recoverInProgressPart(S3Recoverable recoverable) throws IOException {

        final String objectKey = recoverable.incompleteObjectName();
        if (objectKey == null) {
            return Optional.empty();
        }

        // download the file (simple way)
        final RefCountedFileWithStream refCountedFile = tmpFileSupplier.apply(null);
        final File file = refCountedFile.getFile();
        final long numBytes = s3AccessHelper.getObject(objectKey, file);

        if (numBytes != recoverable.incompleteObjectLength()) {
            throw new IOException(
                    String.format(
                            "Error recovering writer: "
                                    + "Downloading the last data chunk file gives incorrect length."
                                    + "File length is %d bytes, RecoveryData indicates %d bytes",
                            numBytes, recoverable.incompleteObjectLength()));
        }

        return Optional.of(file);
    }

    private String pathToObjectName(final Path path) {
        org.apache.hadoop.fs.Path hadoopPath = HadoopFileSystem.toHadoopPath(path);
        if (!hadoopPath.isAbsolute()) {
            hadoopPath = new org.apache.hadoop.fs.Path(fs.getWorkingDirectory(), hadoopPath);
        }

        return hadoopPath.toUri().getScheme() != null && hadoopPath.toUri().getPath().isEmpty()
                ? ""

View on GitHub (pinned to 2f3c205e92)

Solutions

  1. Address the cause reported by the error message: Error recovering writer: Downloading the last data chunk file gives incorrect length.File length is the reported value bytes, RecoveryData indicates the reported value bytes
  2. Verify the inputs, configuration values, and classpath/dependency setup related to this operation, then retry.

Example fix

Correct the condition described ("Error recovering writer: Downloading the last data chunk file gives incorrect length.File length is the reported value bytes, RecoveryData indicates the reported value bytes") and rerun the job or command.

When it happens

Trigger: Triggered at runtime when the operation fails because: Error recovering writer: Downloading the last data chunk file gives incorrect length.File length is the reported value bytes, RecoveryData indicates the reported value bytes.

Common situations: Commonly caused by misconfiguration, missing dependencies or files, unsupported types or operations, or invalid user input leading to: Error recovering writer: Downloading the last data chunk file gives incorrect length.File length is the reported value bytes, RecoveryData indicates the reported value bytes.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/983f85bb9b006836. Report an issue: GitHub.