What causes "Failed to read metadata for blob %s" in apache/flink?

Check with 'gcloud storage stat' / console whether the blob actually exists after the failure — if present it was a consistency/read hiccup

apache/flink · error · IOException

Failed to read metadata for blob %s

Error message

Failed to read metadata for blob %s

What it means

Thrown by GSChecksumWriteChannel.close() when, after successfully closing the underlying write channel, storage.getMetadata(blobIdentifier) returns empty — the freshly written blob's metadata cannot be read back. Since the checksum verification (recommended by Google for streaming uploads) depends on that metadata, the channel cannot confirm the write and fails with IOException.

Source

Thrown at flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/writer/GSChecksumWriteChannel.java:102

        hasher.putBytes(content, start, length);
        return writeChannel.write(content, start, length);
    }

    /**
     * Closes the channel and validates the checksum against the storage. Manually verifying
     * checksums for streaming uploads is recommended by Google, see here:
     * https://cloud.google.com/storage/docs/streaming
     *
     * @throws IOException On underlying failure or non-matching checksums
     */
    public void close() throws IOException {
        LOGGER.trace("Closing write channel to blob {}", blobIdentifier);

        // close channel and get blob metadata
        writeChannel.close();
        Optional<GSBlobStorage.BlobMetadata> blobMetadata = storage.getMetadata(blobIdentifier);
        if (!blobMetadata.isPresent()) {
            throw new IOException(
                    String.format("Failed to read metadata for blob %s", blobIdentifier));
        }

        // make sure checksums match
        String writeChecksum = ChecksumUtils.convertChecksumToString(hasher.hash().asInt());
        String blobChecksum = blobMetadata.get().getChecksum();
        if (!writeChecksum.equals(blobChecksum)) {
            throw new IOException(
                    String.format(
                            "Checksum mismatch writing blob %s: expected %s but found %s",
                            blobIdentifier, writeChecksum, blobChecksum));
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

Check with 'gcloud storage stat' / console whether the blob actually exists after the failure — if present it was a consistency/read hiccup
Retry the operation (recreate the channel and rewrite) — data was either not durably written or is unverifiable
If using a GCS-compatible emulator, ensure it implements object metadata reads consistently
Verify no lifecycle/retention rule deletes objects on creation

Defensive patterns

Strategy: retry

Try / catch

try {
    channel.close();
} catch (java.io.IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("Failed to read metadata")) {
        // blob may still exist; verify then retry the write once
        verifyAndRetryWrite(blobIdentifier);
    } else { throw e; }
}

Prevention

Avoid instant write-then-stat patterns against GCS emulators without verifying their consistency
Ensure no lifecycle rule deletes objects at creation time
Log the blob id on failure and check it in the console before retrying

When it happens

Trigger: Closing a checksummed write channel to gs:// where the metadata read after close returns Optional.empty(): eventual-consistency window in GCS, the blob was deleted immediately after upload, storage adapter misconfiguration, or a getMetadata call that swallowed an underlying error into an empty result.

Common situations: Very rapid write-then-stat cycles in tests/mocked storage; third-party GCS-compatible stores (minio/fake-gcs-server) with weaker consistency; lifecycle rules deleting objects instantly.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/dfb74f9462907699. Report an issue: GitHub.