What causes "Blob %s already exists during attempted commit" in apache/flink?

Make each writer/commit target a unique final blob path (e.g. part files with unique part indexes) so concurrent or retried commits cannot collide

apache/flink · error · IOException

Blob %s already exists during attempted commit

Error message

Blob %s already exists during attempted commit

What it means

GSRecoverableWriterCommitter.commit first checks that the final blob does not already exist, and throws this IOException if it does. The GCS committer implements create-new semantics (see the linked PR discussion): a commit must atomically create the target blob, so an existing blob means the commit is a duplicate or conflicting with existing data rather than an idempotent repeat.

Source

Thrown at flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/writer/GSRecoverableWriterCommitter.java:89

        Preconditions.checkArgument(composeMaxBlobs > 0);
        this.composeMaxBlobs = composeMaxBlobs;
    }

    GSRecoverableWriterCommitter(
            GSBlobStorage storage, GSFileSystemOptions options, GSCommitRecoverable recoverable) {
        this(storage, options, recoverable, BlobUtils.COMPOSE_MAX_BLOBS);
    }

    @Override
    public void commit() throws IOException {
        LOGGER.trace("Committing recoverable with options {}: {}", options, recoverable);

        // see discussion: https://github.com/apache/flink/pull/15599#discussion_r623127365
        // first, make sure the final blob doesn't already exist
        Optional<GSBlobStorage.BlobMetadata> blobMetadata =
                storage.getMetadata(recoverable.finalBlobIdentifier);
        if (blobMetadata.isPresent()) {
            throw new IOException(
                    String.format(
                            "Blob %s already exists during attempted commit",
                            recoverable.finalBlobIdentifier));
        }

        // write the final blob
        writeFinalBlob();

        // clean up after successful commit
        cleanupTemporaryBlobs();
    }

    @Override
    public void commitAfterRecovery() throws IOException {
        LOGGER.trace(
                "Committing recoverable after recovery with options {}: {}", options, recoverable);

        // see discussion: https://github.com/apache/flink/pull/15599#discussion_r623127365

View on GitHub (pinned to 2f3c205e92)

Solutions

Make each writer/commit target a unique final blob path (e.g. part files with unique part indexes) so concurrent or retried commits cannot collide
On recovery, detect that the blob already exists and skip re-committing instead of calling commit() again (treat the earlier commit as done, then clean up temp blobs)
Inspect the existing blob: if it was produced by the same in-flight attempt, remove leftover temporary component blobs and continue without re-committing

Example fix

// before
committer.commit();
// after: guard on recovery
googStorage.delete(finalBlobIdentifier) only if safe, or skip commit when metadata already present
if (storage.getMetadata(recoverable.finalBlobIdentifier).isEmpty()) {
    committer.commit();
}

Defensive patterns

Strategy: validation

Validate before calling

// before committing, check for an existing final blob (idempotent skip)
if (storage.getMetadata(recoverable.finalBlobIdentifier).isPresent()) {
    // previous commit already landed; skip commit, optionally clean temp blobs
    cleanupTemporaryBlobs();
    return;
}
committer.commit();

Try / catch

try {
    committer.commit();
} catch (IOException e) {
    if (e.getMessage().contains("already exists")) {
        // decide: identical prior commit -> treat as success + cleanup; conflict -> fail loudly
    } else {
        throw e;
    }
}

Prevention

Give every writer unique final part-file names so commits cannot collide
Make recovery logic aware that commit may have already succeeded
Always run cleanup of temporary blobs after successful commits

When it happens

Trigger: Calling commit() twice for the same recoverable (e.g. job manager and a recovered task both committing); two jobs/parts writing to the same final blob path; a previous commit succeeded and cleanup was interrupted, then commit is retried; recovery logic that re-commits instead of using commitAfterRecovery-style handling.

Common situations: Task failover replaying the commit step without recovery awareness; misconfigured sink output paths causing multiple writers to target one file; retrying a job from a checkpoint whose pending file was already fully committed and renamed.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/6393870e14db91af. Report an issue: GitHub.