apache/kafka · error · UnsupportedOperationException

Compression is not yet supported with the incremental buffer

Error message

Compression is not yet supported with the incremental buffer.memory allocation strategy

What it means

Thrown by the ChunkedRecordAccumulator constructor when compression is enabled while the incremental buffer.memory allocation strategy is active. The incremental strategy backs batches with fixed-size chunks that grow on demand; the in-place compressor can overshoot its write limit, and the strategy has not yet implemented the mid-record growth fallback, so it rejects any non-NONE compression up front with an UnsupportedOperationException. This is a deliberate, documented gap (the source carries a TODO) rather than a permanent restriction.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/producer/internals/ChunkedRecordAccumulator.java:85

                                    long retryBackoffMs,
                                    long retryBackoffMaxMs,
                                    int deliveryTimeoutMs,
                                    PartitionerConfig partitionerConfig,
                                    Metrics metrics,
                                    String metricGrpName,
                                    Time time,
                                    TransactionManager transactionManager,
                                    BufferPool bufferPool) {
        super(logContext, batchSize, compression, lingerMs, retryBackoffMs, retryBackoffMaxMs,
                deliveryTimeoutMs, partitionerConfig, metrics, metricGrpName, time, transactionManager, bufferPool);
        if (bufferPool.allocationMode() != BufferPool.AllocationMode.INCREMENTAL)
            throw new IllegalArgumentException("bufferPool must serve "
                    + BufferPool.AllocationMode.INCREMENTAL + " allocation, but serves "
                    + bufferPool.allocationMode());
        // TODO: drop this once the incremental strategy supports compressed data (with the
        //   mid-record growth fallback for compressor overshoot).
        if (compression.type() != CompressionType.NONE)
            throw new UnsupportedOperationException(
                    "Compression is not yet supported with the incremental buffer.memory allocation strategy");
        this.chunkedFree = bufferPool;
    }

    public ChunkedRecordAccumulator(LogContext logContext,
                                    int batchSize,
                                    Compression compression,
                                    int lingerMs,
                                    long retryBackoffMs,
                                    long retryBackoffMaxMs,
                                    int deliveryTimeoutMs,
                                    Metrics metrics,
                                    String metricGrpName,
                                    Time time,
                                    TransactionManager transactionManager,
                                    BufferPool bufferPool) {
        this(logContext, batchSize, compression, lingerMs, retryBackoffMs, retryBackoffMaxMs,
                deliveryTimeoutMs, new PartitionerConfig(), metrics, metricGrpName, time, transactionManager,

View on GitHub (pinned to c31c9215e1)

Solutions

1. Disable compression when using the incremental strategy: set compression.type=none (or remove producer.compression.type) if chunked accumulation is more important than wire compression.
2. Keep compression and avoid the incremental strategy: lower batch.size below 16KB so the producer uses the full strategy, which fully supports compression.
3. Track the upstream TODO and upgrade the Kafka client once the incremental strategy adds the compressor mid-record growth fallback, then re-enable compression.

Example fix

// before - incremental strategy selected (batch.size >= 16KB) with compression
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");

// after option A - keep chunked strategy, drop compression
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "none");

// after option B - keep compression, use full strategy
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 8 * 1024);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");

// Compression and incremental buffer.memory allocation are mutually exclusive.
String compression = (String) producerProps.get("compression.type"); // default "none"
String allocationMode = (String) producerProps.get("buffer.memory.allocation"); // hypothetical key
boolean incremental = "incremental".equalsIgnoreCase(allocationMode);
boolean compressed = compression != null && !"none".equalsIgnoreCase(compression);
if (incremental && compressed) {
    throw new IllegalArgumentException(
        "compression.type=" + compression + " is incompatible with incremental buffer.memory allocation");
}

// Thrown at ChunkedRecordAccumulator construction, surfaced when the producer is built.
try {
    KafkaProducer<K, V> producer = new KafkaProducer<>(props);
} catch (UnsupportedOperationException e) {
    if (e.getMessage().contains("Compression is not yet supported")) {
        // Drop incremental allocation mode OR drop compression; then rebuild props.
        props.put("buffer.memory.allocation", "full"); // or unset compression.type
        producer = new KafkaProducer<>(props);
    } else throw e;
}

When it happens

Trigger: Configuring the producer with compression.type set to gzip, snappy, lz4, zstd (or producer.compression.type) while the incremental strategy is selected, which happens when batch.size >= ChunkedRecordAccumulator.CHUNK_SIZE (16KB). The ChunkedRecordAccumulator constructor checks compression.type() != NONE and throws.

Common situations: Adopting the newer incremental buffer.memory strategy (large batch.size) on a producer that already had compression enabled; upgrading the Kafka client to a version that introduced the incremental strategy and finding existing compression configs now trip this check; setting batch.size to 32KB or 64KB for throughput while keeping lz4 compression; copying a producer config template that includes compression into a new high-throughput producer.

Related errors

• Attempt to allocate {totalSize} bytes ({numChunks} chunks of
• Attempt to allocate {size} bytes, but there is a hard limit
• Producer closed while send in progress
• Invalid producer ID and epoch values: {producerId}:{epoch}.
• Must set retries to non-zero when using the idempotent produ

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/af2ff386f6129ef0.json. Report an issue: GitHub.