apache/flink · error · IllegalArgumentException

S3 File System cannot recover recoverable for other file sys

Error message

S3 File System cannot recover recoverable for other file system: {}

What it means

S3RecoverableWriter.castToS3Recoverable throws IllegalArgumentException when a CommitRecoverable that is not an S3Recoverable instance is passed to the writer's recover, commit, or resume operations. The S3 writer can only finish uploads that its own S3RecoverableWriter started, because the recoverable carries S3-specific data (uploadId, part ETags).

Source

Thrown at flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/writer/S3RecoverableWriter.java:139

    @Override
    @SuppressWarnings({"rawtypes", "unchecked"})
    public SimpleVersionedSerializer<ResumeRecoverable> getResumeRecoverableSerializer() {
        return (SimpleVersionedSerializer) S3RecoverableSerializer.INSTANCE;
    }

    @Override
    public boolean supportsResume() {
        return true;
    }

    // --------------------------- Utils ---------------------------

    private static S3Recoverable castToS3Recoverable(CommitRecoverable recoverable) {
        if (recoverable instanceof S3Recoverable) {
            return (S3Recoverable) recoverable;
        }
        throw new IllegalArgumentException(
                "S3 File System cannot recover recoverable for other file system: " + recoverable);
    }

    // --------------------------- Static Constructor ---------------------------

    public static S3RecoverableWriter writer(
            final FileSystem fs,
            final FunctionWithException<File, RefCountedFileWithStream, IOException>
                    tempFileCreator,
            final S3AccessHelper s3AccessHelper,
            final Executor uploadThreadPool,
            final long userDefinedMinPartSize,
            final int maxConcurrentUploadsPerStream) {

        checkArgument(userDefinedMinPartSize >= S3_MULTIPART_MIN_PART_SIZE);

        final S3RecoverableMultipartUploadFactory uploadFactory =
                new S3RecoverableMultipartUploadFactory(

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Ensure the filesystem scheme used to create the RecoverableWriter is identical to the one used when committing; both phases must resolve to the same S3 filesystem.
2. Check that the sink serialized the concrete S3Recoverable (via S3RecoverableSerializer) and not some other CommitRecoverable type.
3. In tests, construct S3Recoverable instances (or use S3RecoverableSerializer round-trips) instead of anonymous CommitRecoverable stubs.
4. If you need to support multiple filesystems, branch on recoverable type before choosing the writer.

Example fix

// before
RecoverableWriter s3Writer = s3Fs.createRecoverableWriter();
s3Writer.recoverForCommit(recoverableFromLocalFs); // IllegalArgumentException

// after
if (recoverable instanceof S3Recoverable) {
    s3Writer.recoverForCommit(recoverable);
} else {
    RecoverableWriter correctWriter = originFs.createRecoverableWriter();
    correctWriter.recoverForCommit(recoverable);
}

private static S3Recoverable asS3Recoverable(CommitRecoverable r) {
    if (r instanceof S3Recoverable) {
        return (S3Recoverable) r;
    }
    return null; // caller routes to the matching writer instead of failing
}

try {
    s3Writer.recoverForCommit(recoverable);
} catch (IllegalArgumentException e) {
    // recoverable belongs to another filesystem: re-acquire the correct
    // RecoverableWriter from the originating FileSystem and retry there
}

When it happens

Trigger: Calling S3RecoverableWriter.recoverForCommit / commit / resume with a CommitRecoverable produced by a different filesystem's RecoverableWriter (e.g. local filesystem, HDFS, or another object store), or with a mock/forged CommitRecoverable object.

Common situations: Job graph rewired to a different output filesystem between the write phase and the commit phase (e.g. scheme changed from s3:// to file:// in config), a sink that stores GenericWriterProperties instead of concrete recoverables, or unit tests passing dummy CommitRecoverable implementations.

Related errors

• Cannot sync state to system like S3. Use persist() to create
• Only S3 to local copies are currently supported: {} -> {}
• Recoverable writer not available
• Input opening request timed out. Opener was {} alive. Stack
• Output path could not be initialized.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/fda8c89a084d7ec2. Report an issue: GitHub.