apache/flink · error · UnsupportedOperationException

Recoverable writers on AzureBlob are only supported for ABFS

Error message

Recoverable writers on AzureBlob are only supported for ABFS

What it means

Thrown by AzureBlobRecoverableWriter.checkSupportedFSSchemes when the Hadoop FileSystem passed to the writer has a scheme other than 'abfs' or 'abfss' (case-insensitive). The recoverable-writer machinery (append, persist, commit-by-rename) is only implemented for the ABFS driver, so legacy wasb/wasbs endpoints or other filesystems are rejected explicitly.

Source

Thrown at flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AzureBlobRecoverableWriter.java:44

import java.io.IOException;

/** Recoverable writer for AzureBlob file system. */
public class AzureBlobRecoverableWriter extends HadoopRecoverableWriter {
    /**
     * Creates a new Recoverable writer.
     *
     * @param fs The AzureBlob file system on which the writer operates.
     */
    public AzureBlobRecoverableWriter(FileSystem fs) {
        super(fs);
    }

    protected void checkSupportedFSSchemes(org.apache.hadoop.fs.FileSystem fs) {
        // This writer is only supported on a subset of file systems
        if (!("abfs".equalsIgnoreCase(fs.getScheme())
                || "abfss".equalsIgnoreCase(fs.getScheme()))) {
            throw new UnsupportedOperationException(
                    "Recoverable writers on AzureBlob are only supported for ABFS");
        }
    }

    @Override
    protected RecoverableFsDataOutputStream getRecoverableFsDataOutputStream(
            org.apache.hadoop.fs.Path targetFile, org.apache.hadoop.fs.Path tempFile)
            throws IOException {
        return new AzureBlobFsRecoverableDataOutputStream(fs, targetFile, tempFile);
    }

    @Override
    public RecoverableFsDataOutputStream recover(ResumeRecoverable recoverable) throws IOException {
        return new AzureBlobFsRecoverableDataOutputStream(fs, (HadoopFsRecoverable) recoverable);
    }

    @Override
    public RecoverableFsDataOutputStream.Committer recoverForCommit(CommitRecoverable recoverable)

View on GitHub (pinned to 2f3c205e92)

Solutions

Change sink/checkpoint/output paths from wasb(s):// to abfs(s):// (e.g. abfs://container@account.dfs.core.windows.net/path)
Use abfss:// for encrypted-channel ABFS endpoints where required
If you must stay on wasb, use a non-recoverable sink (bucketing/file sink without recoverable writer) — expect no exactly-once file guarantees

Example fix

// before
Path path = new Path("wasb://container@account.blob.core.windows.net/out");
FileSink.forRowFormat(path, encoder)
    .build(); // constructs AzureBlobRecoverableWriter -> fails

// after
Path path = new Path("abfs://container@account.dfs.core.windows.net/out");
FileSink.forRowFormat(path, encoder)
    .build();

Defensive patterns

Strategy: validation

Validate before calling

// before creating the sink/writer, assert the scheme is supported
String scheme = path.toUri().getScheme();
if (!("abfs".equalsIgnoreCase(scheme) || "abfss".equalsIgnoreCase(scheme))) {
    throw new IllegalArgumentException(
        "Recoverable Azure sink requires abfs:// or abfss://, got: " + path);
}

Try / catch

try {
    new AzureBlobRecoverableWriter(hadoopFs);
} catch (UnsupportedOperationException e) {
    // migrate the path to abfs(s):// or fall back to a non-recoverable sink
}

Prevention

Migrate legacy wasb:// URIs to abfs(s):// during Flink upgrades
Add a config lint step that rejects wasb paths for recoverable sinks
Remember abfs uses the dfs endpoint host (account.dfs.core.windows.net)

When it happens

Trigger: Creating an AzureBlobRecoverableWriter (e.g. StreamingFileSink/FileSink on 'wasb://...' or 'wasbs://...' paths) — the scheme check runs at writer construction and fails fast before any write.

Common situations: Migrating legacy WASB-based jobs to the recoverable sink without changing URIs; mixing schemes in one path; typo in the scheme.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/0b4e2aa9973412ef. Report an issue: GitHub.