{"record":{"id":"dfb74f9462907699","repo":"apache/flink","slug":"failed-to-read-metadata-for-blob-s","errorCode":null,"errorMessage":"Failed to read metadata for blob %s","messagePattern":"Failed to read metadata for blob (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/writer/GSChecksumWriteChannel.java","lineNumber":102,"sourceCode":"        hasher.putBytes(content, start, length);\n        return writeChannel.write(content, start, length);\n    }\n\n    /**\n     * Closes the channel and validates the checksum against the storage. Manually verifying\n     * checksums for streaming uploads is recommended by Google, see here:\n     * https://cloud.google.com/storage/docs/streaming\n     *\n     * @throws IOException On underlying failure or non-matching checksums\n     */\n    public void close() throws IOException {\n        LOGGER.trace(\"Closing write channel to blob {}\", blobIdentifier);\n\n        // close channel and get blob metadata\n        writeChannel.close();\n        Optional<GSBlobStorage.BlobMetadata> blobMetadata = storage.getMetadata(blobIdentifier);\n        if (!blobMetadata.isPresent()) {\n            throw new IOException(\n                    String.format(\"Failed to read metadata for blob %s\", blobIdentifier));\n        }\n\n        // make sure checksums match\n        String writeChecksum = ChecksumUtils.convertChecksumToString(hasher.hash().asInt());\n        String blobChecksum = blobMetadata.get().getChecksum();\n        if (!writeChecksum.equals(blobChecksum)) {\n            throw new IOException(\n                    String.format(\n                            \"Checksum mismatch writing blob %s: expected %s but found %s\",\n                            blobIdentifier, writeChecksum, blobChecksum));\n        }\n    }\n}\n","sourceCodeStart":84,"sourceCodeEnd":117,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/writer/GSChecksumWriteChannel.java#L84-L117","documentation":"Thrown by GSChecksumWriteChannel.close() when, after successfully closing the underlying write channel, storage.getMetadata(blobIdentifier) returns empty — the freshly written blob's metadata cannot be read back. Since the checksum verification (recommended by Google for streaming uploads) depends on that metadata, the channel cannot confirm the write and fails with IOException.","triggerScenarios":"Closing a checksummed write channel to gs:// where the metadata read after close returns Optional.empty(): eventual-consistency window in GCS, the blob was deleted immediately after upload, storage adapter misconfiguration, or a getMetadata call that swallowed an underlying error into an empty result.","commonSituations":"Very rapid write-then-stat cycles in tests/mocked storage; third-party GCS-compatible stores (minio/fake-gcs-server) with weaker consistency; lifecycle rules deleting objects instantly.","solutions":["Check with 'gcloud storage stat' / console whether the blob actually exists after the failure — if present it was a consistency/read hiccup","Retry the operation (recreate the channel and rewrite) — data was either not durably written or is unverifiable","If using a GCS-compatible emulator, ensure it implements object metadata reads consistently","Verify no lifecycle/retention rule deletes objects on creation"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    channel.close();\n} catch (java.io.IOException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"Failed to read metadata\")) {\n        // blob may still exist; verify then retry the write once\n        verifyAndRetryWrite(blobIdentifier);\n    } else { throw e; }\n}","preventionTips":["Avoid instant write-then-stat patterns against GCS emulators without verifying their consistency","Ensure no lifecycle rule deletes objects at creation time","Log the blob id on failure and check it in the console before retrying"],"tags":["gcs","checksum","metadata","consistency","write"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}