apache/flink · error · IllegalStateException

Streaming mode not support overwrite.

Error message

Streaming mode not support overwrite.

What it means

FileSystemTableSink.consume checks the execution mode. If the source is unbounded (streaming, sinkContext.isBounded() returns false) and overwrite mode is enabled, it throws an IllegalStateException. Overwrite mode replaces all existing data in the output directory, which is only meaningful and safe in batch (bounded) execution. In streaming, data continuously arrives so overwrite semantics are undefined.

Source

Thrown at flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSink.java:158

            @Override
            public DataStreamSink<?> consumeDataStream(
                    ProviderContext providerContext, DataStream<RowData> dataStream) {
                return consume(providerContext, dataStream, sinkContext);
            }
        };
    }

    private DataStreamSink<?> consume(
            ProviderContext providerContext, DataStream<RowData> dataStream, Context sinkContext) {
        final int inputParallelism = dataStream.getParallelism();
        final int parallelism = Optional.ofNullable(configuredParallelism).orElse(inputParallelism);
        boolean parallelismConfigued = configuredParallelism != null;

        if (sinkContext.isBounded()) {
            return createBatchSink(dataStream, sinkContext, parallelism, parallelismConfigued);
        } else {
            if (overwrite) {
                throw new IllegalStateException("Streaming mode not support overwrite.");
            }

            return createStreamingSink(
                    providerContext, dataStream, sinkContext, parallelism, parallelismConfigued);
        }
    }

    private RowDataPartitionComputer partitionComputer() {
        return new RowDataPartitionComputer(
                defaultPartName,
                DataType.getFieldNames(physicalRowDataType).toArray(new String[0]),
                DataType.getFieldDataTypes(physicalRowDataType).toArray(new DataType[0]),
                partitionKeys.toArray(new String[0]));
    }

    private DataStreamSink<RowData> createBatchSink(
            DataStream<RowData> inputStream,
            Context sinkContext,

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Use BATCH execution mode if overwrite is needed: set execution.runtime-mode = BATCH.
2. Remove the overwrite flag if streaming execution is required — use INSERT INTO instead of INSERT OVERWRITE.
3. Ensure the input source is bounded (e.g. FileSource in batch mode) when using overwrite.

Example fix

-- before (streaming + overwrite -> fails)
SET 'execution.runtime-mode' = 'streaming';
INSERT OVERWRITE sink_t SELECT * FROM kafka_source;

-- after
SET 'execution.runtime-mode' = 'batch';
INSERT OVERWRITE sink_t SELECT * FROM batch_source;

// Before using overwrite on a filesystem sink, verify execution mode
boolean isStreaming = !sinkContext.isBounded();
if (overwrite && isStreaming) {
    throw new IllegalStateException(
        "Overwrite mode is not supported in streaming execution. Use BATCH mode.");
}

When it happens

Trigger: The table sink is configured with overwrite=true (via the DynamicTableSink API or factory) and the job runs in STREAMING mode. The overwrite flag is set when consuming an unbounded (continuous) data stream.

Common situations: A batch pipeline that was originally bounded is changed to streaming without removing overwrite. Using INSERT OVERWRITE semantics on a filesystem sink with an unbounded source (e.g. Kafka). SQL INSERT OVERWRITE on a streaming table.

Related errors

• Could not find any format factory for identifier '%s' in the
• Compaction reader not support DataStructure converter.
• Can not find format factory.
• Currently, filesystem sink doesn't support setting paralleli
• Output path '{}' could not be initialized. Canceling task...

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/7ab952506b67485a. Report an issue: GitHub.