apache/flink · error · UnsupportedOperationException

Bulk Part Writers do not support "pause and resume" operatio

Error message

Bulk Part Writers do not support "pause and resume" operations.

What it means

Thrown by BulkPartWriter.persist() because bulk-encoding writers (Parquet, ORC, Avro bulk) cannot snapshot their in-progress state for later recovery. The BulkWriter interface is append-only with no mid-stream checkpoint capability, so pausing and resuming an in-progress bulk file is impossible. This is a fundamental limitation of bulk formats, not a configuration oversight.

Source

Thrown at flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/BulkPartWriter.java:57

            final BucketID bucketId,
            final Path path,
            final RecoverableFsDataOutputStream currentPartStream,
            final BulkWriter<IN> writer,
            final long creationTime) {
        super(bucketId, path, currentPartStream, creationTime);
        this.writer = Preconditions.checkNotNull(writer);
    }

    @Override
    public void write(IN element, long currentTime) throws IOException {
        ensureWriteType(Type.RECORD_WISE);
        writer.addElement(element);
        markWrite(currentTime);
    }

    @Override
    public InProgressFileRecoverable persist() {
        throw new UnsupportedOperationException(
                "Bulk Part Writers do not support \"pause and resume\" operations.");
    }

    @Override
    public PendingFileRecoverable closeForCommit() throws IOException {
        writer.flush();
        writer.finish();
        return super.closeForCommit();
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

Use a RollingPolicy that rolls the in-progress file to pending at checkpoint rather than persisting it — e.g. configure the policy so that in-progress files are committed (closeForCommit) rather than persisted.
Switch to FileSink (the newer API) which handles bulk format checkpointing by rolling in-progress files on checkpoint.
If using StreamingFileSink, ensure the bucket rolling policy does not call persist on bulk writers — this typically means setting the in-progress rolling behavior correctly.

Example fix

// before — StreamingFileSink bulk format with persist-on-checkpoint
StreamingFileSink.forBulkFormat(path, parquetBulkWriterFactory)
    .withRollingPolicy(new OnCheckpointRollingPolicy<>()) // may trigger persist()
    .build();
// after — use FileSink which rolls in-progress bulk files on checkpoint
FileSink.forBulkFormat(OutputFileConfig, parquetBulkWriterFactory)
    .withRollingPolicy(OnCheckpointRollingPolicy.build())
    .build();

Defensive patterns

Strategy: validation

Validate before calling

// Before using bulk format, ensure checkpoint policy rolls rather than persists
if (writer instanceof BulkPartWriter) {
    // persist() is unsupported; ensure the rolling policy commits on checkpoint
    // rather than calling persist()
    Objects.requireNonNull(rollingPolicy, "Rolling policy required for bulk format");
}

Prevention

Prefer FileSink over StreamingFileSink for bulk formats — it handles checkpoint rolling correctly.
Never call persist() on a BulkPartWriter; use closeForCommit() to roll the file.
Test checkpoint/recovery behavior with your specific bulk format before production.

When it happens

Trigger: The File Sink / StreamingFileSink in BULK format mode triggers persist() on an in-progress file during checkpointing — e.g. when the checkpoint policy requires preserving in-progress files across recovery.

Common situations: Using StreamingFileSink.forBulkFormat() with a RollingPolicy that attempts to persist in-progress files at checkpoint time; or a custom BucketAssigner/OutputFileConfig that indirectly triggers persist on bulk writers.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/f583e525a67ab9a4. Report an issue: GitHub.