{"record":{"id":"3daaec31c55a2d7f","repo":"apache/flink","slug":"checksum-mismatch-writing-blob-s-expected-s-but","errorCode":null,"errorMessage":"Checksum mismatch writing blob %s: expected %s but found %s","messagePattern":"Checksum mismatch writing blob (.+?): expected (.+?) but found (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/writer/GSChecksumWriteChannel.java","lineNumber":110,"sourceCode":"     *\n     * @throws IOException On underlying failure or non-matching checksums\n     */\n    public void close() throws IOException {\n        LOGGER.trace(\"Closing write channel to blob {}\", blobIdentifier);\n\n        // close channel and get blob metadata\n        writeChannel.close();\n        Optional<GSBlobStorage.BlobMetadata> blobMetadata = storage.getMetadata(blobIdentifier);\n        if (!blobMetadata.isPresent()) {\n            throw new IOException(\n                    String.format(\"Failed to read metadata for blob %s\", blobIdentifier));\n        }\n\n        // make sure checksums match\n        String writeChecksum = ChecksumUtils.convertChecksumToString(hasher.hash().asInt());\n        String blobChecksum = blobMetadata.get().getChecksum();\n        if (!writeChecksum.equals(blobChecksum)) {\n            throw new IOException(\n                    String.format(\n                            \"Checksum mismatch writing blob %s: expected %s but found %s\",\n                            blobIdentifier, writeChecksum, blobChecksum));\n        }\n    }\n}\n","sourceCodeStart":92,"sourceCodeEnd":117,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/writer/GSChecksumWriteChannel.java#L92-L117","documentation":"Thrown by GSChecksumWriteChannel.close() when the CRC32C checksum computed locally over the written bytes does not match the checksum GCS reports for the uploaded blob. This is strong evidence the blob stored in GCS differs from what was written — corruption in transit, a partial upload, or two writers hitting the same object.","triggerScenarios":"Closing a checksummed GS write channel where the local hasher's CRC32C != blob metadata checksum: interrupted/resumable upload leaving mixed generations, concurrent writes to the same gs:// object from another process, or corrupt payload from a broken proxy/serializer.","commonSituations":"Two jobs or a job plus an external uploader writing the same object name; network middleboxes mangling payloads; GCS composite-object quirks with certain upload modes.","solutions":["Ensure object names written by the channel are unique per attempt (include subtask/attempt counters as the file sink already does)","Rewrite the object: delete the corrupted blob and rerun the failed write","Compare the expected/found checksums from the message against 'gcloud storage hash' output to confirm corruption","If reproducible, capture a network trace or test without proxies between the TaskManager and storage.googleapis.com"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    channel.close();\n} catch (java.io.IOException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"Checksum mismatch\")) {\n        // stored blob is corrupt: delete and rewrite with a fresh unique name\n        deleteBlob(blobIdentifier);\n        rewriteWithUniqueName();\n    } else { throw e; }\n}","preventionTips":["Make object names unique per write attempt so retries never mix generations","Ensure no second writer targets the same gs object concurrently","Capture expected vs found checksums from the message for postmortems"],"tags":["gcs","checksum","data-corruption","write","consistency"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}