apache/flink · error · UnsupportedOperationException

Cannot sync state to system like S3. Use persist() to create

Error message

Cannot sync state to system like S3. Use persist() to create a persistent recoverable intermediate point.

What it means

RefCountedBufferingFileStream.sync() deliberately refuses to sync to an object store like S3. Object stores are eventually consistent and cannot be fsync'd like a local file; the API contract here is to call persist() which creates a persistent recoverable intermediate point instead.

Source

Thrown at flink-core/src/main/java/org/apache/flink/core/fs/RefCountedBufferingFileStream.java:111

        if (len > buffer.length - positionInBuffer) {
            flush();
        }

        System.arraycopy(b, off, buffer, positionInBuffer, len);
        positionInBuffer += len;
    }

    @Override
    public void flush() throws IOException {
        currentTmpFile.write(buffer, 0, positionInBuffer);
        currentTmpFile.flush();
        positionInBuffer = 0;
    }

    @Override
    public void sync() throws IOException {
        throw new UnsupportedOperationException(
                "Cannot sync state to system like S3. "
                        + "Use persist() to create a persistent recoverable intermediate point.");
    }

    @Override
    public boolean isClosed() throws IOException {
        return closed;
    }

    @Override
    public void close() {
        if (!closed) {
            currentTmpFile.closeStream();
            closed = true;
        }
    }

    @Override

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Replace the sync() call with persist() to get a persistent recoverable intermediate point.
2. If using a recoverable writer (e.g. for S3), use the writer's persistForRecoverySafely()/persist() path instead of sync().
3. Guard the call: check `instanceof RefCountedBufferingFileStream` or check the filesystem is not object-store before sync().
4. Re-evaluate whether sync semantics are required at all for the target filesystem.

Example fix

// before
stream.sync();

// after (recoverable writer path)
RecoverableWriter.CommitRecoverable recoverable = stream.persist();

static boolean supportsSync(FSDataOutputStream out) {
    return !(out instanceof RefCountedBufferingFileStream);
}

try {
    stream.sync();
} catch (UnsupportedOperationException e) {
    if (stream instanceof RefCountedBufferingFileStream) {
        recoverable = ((RecoverableWriter) writer).persist();
    } else { throw e; }
}

When it happens

Trigger: Calling `sync()` (the FSDataOutputStream/SyncableDataOutputStream interface method) on a RefCountedBufferingFileStream, which is the buffering stream backing Flink's recoverable S3-style sink writer.

Common situations: Custom sink writers or test code that assumes a local filesystem contract and calls sync(); checkpointing against an S3-backed target where a generic path calls Syncable.sync().

Related errors

• Input opening request timed out. Opener was {} alive. Stack
• File Not Found: {}
• Missing data in tmp file: {}
• Cannot clean commit: File has trailing junk data.
• Committing file failed, could not rename {} -> {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/9d2b931e17e0822c. Report an issue: GitHub.