apache/flink · error · UnsupportedOperationException

Recoverable writer not available

Error message

Recoverable writer not available

What it means

NativeS3FileSystem.createRecoverableWriter throws UnsupportedOperationException when s3AccessHelper is null. The access helper (object operations for multipart uploads) is optional in this filesystem; without it there is no way to implement recoverable (resumable, exactly-once) writes, so the method fails fast rather than returning a broken writer.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3FileSystem.java:537

    @Override
    public void copyFiles(
            List<CopyRequest> requests,
            org.apache.flink.core.fs.ICloseableRegistry closeableRegistry)
            throws IOException {
        checkNotClosed();
        if (bulkCopyHelper == null) {
            throw new UnsupportedOperationException(
                    "Bulk copy not enabled. Set s3.bulk-copy.enabled=true");
        }
        bulkCopyHelper.copyFiles(requests, closeableRegistry);
    }

    @Override
    public RecoverableWriter createRecoverableWriter() throws IOException {
        checkNotClosed();
        if (s3AccessHelper == null) {
            throw new UnsupportedOperationException("Recoverable writer not available");
        }
        return NativeS3RecoverableWriter.writer(
                s3AccessHelper, localTmpDir, s3uploadPartSize, maxConcurrentUploadsPerStream);
    }

    @Override
    public CompletableFuture<Void> closeAsync() {
        if (!closed.compareAndSet(false, true)) {
            return CompletableFuture.completedFuture(null);
        }

        LOG.info("Starting async close of Native S3 FileSystem for bucket: {}", bucketName);
        CompletableFuture<Void> closeFuture =
                CompletableFuture.runAsync(
                                () ->
                                        LOG.info(
                                                "Native S3 FileSystem closed for bucket: {}",
                                                bucketName))

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Construct/obtain the filesystem through the standard factory so the S3AccessHelper is created (ensure credentials and upload settings are configured).
2. If you only need plain writes, use create(...) streams instead of a RecoverableWriter.
3. For sinks requiring exactly-once file commits on S3, verify the filesystem plugin (flink-s3-fs-native shaded jar) is properly installed under plugins/ or lib/.

Example fix

// before
RecoverableWriter w = readOnlyS3Fs.createRecoverableWriter(); // throws

// after
// obtain the filesystem via the factory with full configuration:
FileSystem fs = FileSystem.get(new Path("s3://bucket/").toUri());
RecoverableWriter w = fs.createRecoverableWriter();

static boolean canCreateRecoverableWriter(FileSystem fs) {
    // NativeS3FileSystem only supports it when constructed with the S3 access helper;
    // probe cheaply and catch UnsupportedOperationException on first call at startup
    try {
        fs.createRecoverableWriter();
        return true;
    } catch (UnsupportedOperationException e) {
        return false;
    }
}

try {
    RecoverableWriter writer = fs.createRecoverableWriter();
} catch (UnsupportedOperationException e) {
    // filesystem built without upload support: use plain create() streams or
    // re-obtain the filesystem via the standard factory with full S3 config
}

When it happens

Trigger: Calling createRecoverableWriter() on a NativeS3FileSystem instance constructed without an S3AccessHelper — e.g. a filesystem built for read-only or bulk-copy-only usage, or a factory configuration path that skips upload support.

Common situations: Exactly-once sinks (files sink with exactly-once commit, recoverable writers) configured against a filesystem variant that was constructed without upload capabilities; misconfigured factory wiring in embedded/tests setups.

Related errors

• The configuration is unmodifiable; its contents cannot be ch
• Cannot sync state to system like S3. Use persist() to create
• S3 File System cannot recover recoverable for other file sys
• Invalid assume-role.session-duration '%s' for bucket '%s'. M
• Invalid path-style-access '%s' for bucket '%s'. Must be 'tru

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/0a3d9030fe2d364d. Report an issue: GitHub.