apache/flink · error · IOException

File already exists: {}

Error message

File already exists: {}

What it means

NativeS3FileSystem.create with WriteMode.NO_OVERWRITE first checks exists(path) and throws IOException('File already exists: <path>') if an object already exists at that key. This implements the no-clobber contract required by Flink's FileSystem API and output committers that must not overwrite committed results.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3FileSystem.java:438

     *
     * <p>If explicit directory markers are needed, consider using a custom implementation.
     *
     * @return always returns true (S3 doesn't require explicit directory creation)
     */
    @Override
    public boolean mkdirs(Path path) throws IOException {
        checkNotClosed();
        LOG.debug("mkdirs called for {} - S3 doesn't require explicit directory creation", path);
        return true;
    }

    @Override
    public FSDataOutputStream create(Path path, WriteMode overwriteMode) throws IOException {
        checkNotClosed();
        if (overwriteMode == WriteMode.NO_OVERWRITE) {
            try {
                if (exists(path)) {
                    throw new IOException("File already exists: " + path);
                }
            } catch (FileNotFoundException ignored) {
            }
        } else {
            try {
                delete(path, false);
            } catch (FileNotFoundException ignored) {
            }
        }

        final String key = NativeS3ObjectOperations.extractKey(path);
        return new NativeS3OutputStream(
                clientProvider.getS3Client(),
                bucketName,
                key,
                localTmpDir,
                clientProvider.getEncryptionConfig());
    }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Delete or move the existing object/directory before re-running: s3Fs.delete(path, true).
2. Use WriteMode.OVERWRITE when clobbering is acceptable (create then internally deletes the existing object).
3. Make output filenames unique per attempt (e.g. include subtask index and attempt number) so retries never collide.

Example fix

// before
FSDataOutputStream out = s3Fs.create(path, WriteMode.NO_OVERWRITE);

// after
FSDataOutputStream out = s3Fs.create(path, WriteMode.OVERWRITE);
// or before creating:
if (s3Fs.exists(path)) {
    s3Fs.delete(path, false);
}

// pre-check when NO_OVERWRITE semantics matter
if (s3Fs.exists(path)) {
    throw new IOException("Output already exists: " + path + " — clean up or use unique names");
}
FSDataOutputStream out = s3Fs.create(path, WriteMode.NO_OVERWRITE);

try {
    out = s3Fs.create(path, WriteMode.NO_OVERWRITE);
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().startsWith("File already exists")) {
        // collision: choose a unique name (add attempt id) or delete old output first
    } else {
        throw e;
    }
}

When it happens

Trigger: Calling create(path, WriteMode.NO_OVERWRITE) (or an API defaulting to no-overwrite) when the S3 key already exists — e.g. re-running a job without cleanup, task retry writing the same part file, or output directory reuse.

Common situations: Re-submitting a failed job against the same output prefix without deleting previous part files; concurrent tasks computing identical output filenames; recovery replaying a write after the object was already committed.

Related errors

• Failed to abort multipart upload for key: %s, uploadId: %s
• Input opening request timed out. Opener was {} alive. Stack
• Output path could not be initialized.
• Cannot sync state to system like S3. Use persist() to create
• File already exists: {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/193b49938ce5ca80. Report an issue: GitHub.