{"record":{"id":"f583e525a67ab9a4","repo":"apache/flink","slug":"bulk-part-writers-do-not-support-pause-and-resume","errorCode":null,"errorMessage":"Bulk Part Writers do not support \"pause and resume\" operations.","messagePattern":"Bulk Part Writers do not support \"pause and resume\" operations\\.","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/BulkPartWriter.java","lineNumber":57,"sourceCode":"            final BucketID bucketId,\n            final Path path,\n            final RecoverableFsDataOutputStream currentPartStream,\n            final BulkWriter<IN> writer,\n            final long creationTime) {\n        super(bucketId, path, currentPartStream, creationTime);\n        this.writer = Preconditions.checkNotNull(writer);\n    }\n\n    @Override\n    public void write(IN element, long currentTime) throws IOException {\n        ensureWriteType(Type.RECORD_WISE);\n        writer.addElement(element);\n        markWrite(currentTime);\n    }\n\n    @Override\n    public InProgressFileRecoverable persist() {\n        throw new UnsupportedOperationException(\n                \"Bulk Part Writers do not support \\\"pause and resume\\\" operations.\");\n    }\n\n    @Override\n    public PendingFileRecoverable closeForCommit() throws IOException {\n        writer.flush();\n        writer.finish();\n        return super.closeForCommit();\n    }\n}\n","sourceCodeStart":39,"sourceCodeEnd":68,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/BulkPartWriter.java#L39-L68","documentation":"Thrown by BulkPartWriter.persist() because bulk-encoding writers (Parquet, ORC, Avro bulk) cannot snapshot their in-progress state for later recovery. The BulkWriter interface is append-only with no mid-stream checkpoint capability, so pausing and resuming an in-progress bulk file is impossible. This is a fundamental limitation of bulk formats, not a configuration oversight.","triggerScenarios":"The File Sink / StreamingFileSink in BULK format mode triggers persist() on an in-progress file during checkpointing — e.g. when the checkpoint policy requires preserving in-progress files across recovery.","commonSituations":"Using StreamingFileSink.forBulkFormat() with a RollingPolicy that attempts to persist in-progress files at checkpoint time; or a custom BucketAssigner/OutputFileConfig that indirectly triggers persist on bulk writers.","solutions":["Use a RollingPolicy that rolls the in-progress file to pending at checkpoint rather than persisting it — e.g. configure the policy so that in-progress files are committed (closeForCommit) rather than persisted.","Switch to FileSink (the newer API) which handles bulk format checkpointing by rolling in-progress files on checkpoint.","If using StreamingFileSink, ensure the bucket rolling policy does not call persist on bulk writers — this typically means setting the in-progress rolling behavior correctly."],"exampleFix":"// before — StreamingFileSink bulk format with persist-on-checkpoint\nStreamingFileSink.forBulkFormat(path, parquetBulkWriterFactory)\n    .withRollingPolicy(new OnCheckpointRollingPolicy<>()) // may trigger persist()\n    .build();\n// after — use FileSink which rolls in-progress bulk files on checkpoint\nFileSink.forBulkFormat(OutputFileConfig, parquetBulkWriterFactory)\n    .withRollingPolicy(OnCheckpointRollingPolicy.build())\n    .build();","handlingStrategy":"validation","validationCode":"// Before using bulk format, ensure checkpoint policy rolls rather than persists\nif (writer instanceof BulkPartWriter) {\n    // persist() is unsupported; ensure the rolling policy commits on checkpoint\n    // rather than calling persist()\n    Objects.requireNonNull(rollingPolicy, \"Rolling policy required for bulk format\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer FileSink over StreamingFileSink for bulk formats — it handles checkpoint rolling correctly.","Never call persist() on a BulkPartWriter; use closeForCommit() to roll the file.","Test checkpoint/recovery behavior with your specific bulk format before production."],"tags":["file-sink","bulk-writer","checkpoint","internal-api"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}