apache/flink · error · UnsupportedOperationException

Bulk Part Writers do not support "pause and resume"…

Error message

Bulk Part Writers do not support "pause and resume" operations.

What it means

Thrown by BulkPartWriter.persist() because bulk-encoding writers (Parquet, ORC, Avro bulk) cannot snapshot their in-progress state for later recovery. The BulkWriter interface is append-only with no mid-stream checkpoint capability, so pausing and resuming an in-progress bulk file is impossible. This is a fundamental limitation of bulk formats, not a configuration oversight.

Solutions

Use a RollingPolicy that rolls the in-progress file to pending at checkpoint rather than persisting it — e.g. configure the policy so that in-progress files are committed (closeForCommit) rather than persisted.
Switch to FileSink (the newer API) which handles bulk format checkpointing by rolling in-progress files on checkpoint.
If using StreamingFileSink, ensure the bucket rolling policy does not call persist on bulk writers — this typically means setting the in-progress rolling behavior correctly.

Example fix

// before — StreamingFileSink bulk format with persist-on-checkpoint
StreamingFileSink.forBulkFormat(path, parquetBulkWriterFactory)
    .withRollingPolicy(new OnCheckpointRollingPolicy<>()) // may trigger persist()
    .build();
// after — use FileSink which rolls in-progress bulk files on checkpoint
FileSink.forBulkFormat(OutputFileConfig, parquetBulkWriterFactory)
    .withRollingPolicy(OnCheckpointRollingPolicy.build())
    .build();

Defensive patterns

Strategy: validation

Validate before calling

// Before using bulk format, ensure checkpoint policy rolls rather than persists
if (writer instanceof BulkPartWriter) {
    // persist() is unsupported; ensure the rolling policy commits on checkpoint
    // rather than calling persist()
    Objects.requireNonNull(rollingPolicy, "Rolling policy required for bulk format");
}

Prevention

Prefer FileSink over StreamingFileSink for bulk formats — it handles checkpoint rolling correctly.
Never call persist() on a BulkPartWriter; use closeForCommit() to roll the file.
Test checkpoint/recovery behavior with your specific bulk format before production.

When it happens

Trigger: The File Sink / StreamingFileSink in BULK format mode triggers persist() on an in-progress file during checkpointing — e.g. when the checkpoint policy requires preserving in-progress files across recovery.

Common situations: Using StreamingFileSink.forBulkFormat() with a RollingPolicy that attempts to persist in-progress files at checkpoint time; or a custom BucketAssigner/OutputFileConfig that indirectly triggers persist on bulk writers.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/f583e525a67ab9a4. Report an issue: GitHub.

Appendix: source

Thrown at flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/BulkPartWriter.java:57

            final BucketID bucketId,
            final Path path,
            final RecoverableFsDataOutputStream currentPartStream,
            final BulkWriter<IN> writer,
            final long creationTime) {
        super(bucketId, path, currentPartStream, creationTime);
        this.writer = Preconditions.checkNotNull(writer);
    }

    @Override
    public void write(IN element, long currentTime) throws IOException {
        ensureWriteType(Type.RECORD_WISE);
        writer.addElement(element);
        markWrite(currentTime);
    }

    @Override
    public InProgressFileRecoverable persist() {
        throw new UnsupportedOperationException(
                "Bulk Part Writers do not support \"pause and resume\" operations.");
    }

    @Override
    public PendingFileRecoverable closeForCommit() throws IOException {
        writer.flush();
        writer.finish();
        return super.closeForCommit();
    }
}

View on GitHub (pinned to 2f3c205e92)