{"record":{"id":"9d2b931e17e0822c","repo":"apache/flink","slug":"cannot-sync-state-to-system-like-s3-use-persist","errorCode":null,"errorMessage":"Cannot sync state to system like S3. Use persist() to create a persistent recoverable intermediate point.","messagePattern":"Cannot sync state to system like S3\\. Use persist\\(\\) to create a persistent recoverable intermediate point\\.","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"flink-core/src/main/java/org/apache/flink/core/fs/RefCountedBufferingFileStream.java","lineNumber":111,"sourceCode":"\n        if (len > buffer.length - positionInBuffer) {\n            flush();\n        }\n\n        System.arraycopy(b, off, buffer, positionInBuffer, len);\n        positionInBuffer += len;\n    }\n\n    @Override\n    public void flush() throws IOException {\n        currentTmpFile.write(buffer, 0, positionInBuffer);\n        currentTmpFile.flush();\n        positionInBuffer = 0;\n    }\n\n    @Override\n    public void sync() throws IOException {\n        throw new UnsupportedOperationException(\n                \"Cannot sync state to system like S3. \"\n                        + \"Use persist() to create a persistent recoverable intermediate point.\");\n    }\n\n    @Override\n    public boolean isClosed() throws IOException {\n        return closed;\n    }\n\n    @Override\n    public void close() {\n        if (!closed) {\n            currentTmpFile.closeStream();\n            closed = true;\n        }\n    }\n\n    @Override","sourceCodeStart":93,"sourceCodeEnd":129,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/core/fs/RefCountedBufferingFileStream.java#L93-L129","documentation":"RefCountedBufferingFileStream.sync() deliberately refuses to sync to an object store like S3. Object stores are eventually consistent and cannot be fsync'd like a local file; the API contract here is to call persist() which creates a persistent recoverable intermediate point instead.","triggerScenarios":"Calling `sync()` (the FSDataOutputStream/SyncableDataOutputStream interface method) on a RefCountedBufferingFileStream, which is the buffering stream backing Flink's recoverable S3-style sink writer.","commonSituations":"Custom sink writers or test code that assumes a local filesystem contract and calls sync(); checkpointing against an S3-backed target where a generic path calls Syncable.sync().","solutions":["Replace the sync() call with persist() to get a persistent recoverable intermediate point.","If using a recoverable writer (e.g. for S3), use the writer's persistForRecoverySafely()/persist() path instead of sync().","Guard the call: check `instanceof RefCountedBufferingFileStream` or check the filesystem is not object-store before sync().","Re-evaluate whether sync semantics are required at all for the target filesystem."],"exampleFix":"// before\nstream.sync();\n\n// after (recoverable writer path)\nRecoverableWriter.CommitRecoverable recoverable = stream.persist();","handlingStrategy":"type-guard","validationCode":null,"typeGuard":"static boolean supportsSync(FSDataOutputStream out) {\n    return !(out instanceof RefCountedBufferingFileStream);\n}","tryCatchPattern":"try {\n    stream.sync();\n} catch (UnsupportedOperationException e) {\n    if (stream instanceof RefCountedBufferingFileStream) {\n        recoverable = ((RecoverableWriter) writer).persist();\n    } else { throw e; }\n}","preventionTips":["Never assume sync() works on object-store-backed streams; prefer persist().","Branch sink behavior by filesystem kind (local vs object store).","Document sync semantics at the writer boundary for custom sinks."],"tags":["filesystem","s3","object-store","sync","recoverable-writer"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}