apache/flink · critical · IOException

Checksum mismatch writing blob %s: expected %s but found %s

Error message

Checksum mismatch writing blob %s: expected %s but found %s

What it means

Thrown by GSChecksumWriteChannel.close() when the CRC32C checksum computed locally over the written bytes does not match the checksum GCS reports for the uploaded blob. This is strong evidence the blob stored in GCS differs from what was written — corruption in transit, a partial upload, or two writers hitting the same object.

Source

Thrown at flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/writer/GSChecksumWriteChannel.java:110

     *
     * @throws IOException On underlying failure or non-matching checksums
     */
    public void close() throws IOException {
        LOGGER.trace("Closing write channel to blob {}", blobIdentifier);

        // close channel and get blob metadata
        writeChannel.close();
        Optional<GSBlobStorage.BlobMetadata> blobMetadata = storage.getMetadata(blobIdentifier);
        if (!blobMetadata.isPresent()) {
            throw new IOException(
                    String.format("Failed to read metadata for blob %s", blobIdentifier));
        }

        // make sure checksums match
        String writeChecksum = ChecksumUtils.convertChecksumToString(hasher.hash().asInt());
        String blobChecksum = blobMetadata.get().getChecksum();
        if (!writeChecksum.equals(blobChecksum)) {
            throw new IOException(
                    String.format(
                            "Checksum mismatch writing blob %s: expected %s but found %s",
                            blobIdentifier, writeChecksum, blobChecksum));
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Ensure object names written by the channel are unique per attempt (include subtask/attempt counters as the file sink already does)
2. Rewrite the object: delete the corrupted blob and rerun the failed write
3. Compare the expected/found checksums from the message against 'gcloud storage hash' output to confirm corruption
4. If reproducible, capture a network trace or test without proxies between the TaskManager and storage.googleapis.com

try {
    channel.close();
} catch (java.io.IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("Checksum mismatch")) {
        // stored blob is corrupt: delete and rewrite with a fresh unique name
        deleteBlob(blobIdentifier);
        rewriteWithUniqueName();
    } else { throw e; }
}

When it happens

Trigger: Closing a checksummed GS write channel where the local hasher's CRC32C != blob metadata checksum: interrupted/resumable upload leaving mixed generations, concurrent writes to the same gs:// object from another process, or corrupt payload from a broken proxy/serializer.

Common situations: Two jobs or a job plus an external uploader writing the same object name; network middleboxes mangling payloads; GCS composite-object quirks with certain upload modes.

Related errors

• Failed to read metadata for blob %s
• Error while waiting for job to be initialized
• Translator {} cannot translate the given pipeline {}.
• Could not build the program from JAR file: {}
• Could not cancel job {}.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/3daaec31c55a2d7f. Report an issue: GitHub.