apache/kafka · error · ConfigException

The ${ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCRE

Error message

The ${ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL} ${ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG} does not support compression yet. ${ProducerConfig.COMPRESSION_TYPE_CONFIG} must be set to none.

What it means

Thrown during KafkaProducer construction when the chunked/incremental buffer allocation path is selected (buffer.memory.allocation.strategy=incremental AND batch.size >= ChunkedRecordAccumulator.CHUNK_SIZE) but compression is anything other than NONE. The chunked accumulator does not yet implement compression (tracked by KAFKA-20579), so the two together are an unsupported combination. The check is performed after the 'useIncremental' decision logic and aborts producer startup with a ConfigException.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:477

            int batchSize = Math.max(1, config.getInt(ProducerConfig.BATCH_SIZE_CONFIG));
            String allocationStrategy = config.getString(ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG)
                    .toLowerCase(Locale.ROOT);
            boolean incremental = ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL.equals(allocationStrategy);
            // Use the chunked path only when a batch is at least one full chunk
            // (batch.size >= CHUNK_SIZE). Below that, a batch can't fill even one chunk, so chunking
            // would over-reserve and the producer falls back to the full strategy instead.
            boolean useIncremental = incremental && batchSize >= ChunkedRecordAccumulator.CHUNK_SIZE;
            if (incremental && !useIncremental) {
                log.warn("Ignoring {}={} and falling back to {}: {} is {} bytes, below the {} byte chunk size, " +
                                "so a batch cannot fill a single chunk.",
                        ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG,
                        ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL,
                        ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_FULL,
                        ProducerConfig.BATCH_SIZE_CONFIG, batchSize, ChunkedRecordAccumulator.CHUNK_SIZE);
            }
            // The chunked path does not support compression yet (TODO: KAFKA-20579)
            if (useIncremental && compression.type() != CompressionType.NONE) {
                throw new ConfigException("The " + ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL
                        + " " + ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG
                        + " does not support compression yet. " + ProducerConfig.COMPRESSION_TYPE_CONFIG
                        + " must be set to none.");
            }
            if (useIncremental) {
                this.accumulator = new ChunkedRecordAccumulator(logContext,
                        batchSize,
                        compression,
                        lingerMs(config),
                        retryBackoffMs,
                        retryBackoffMaxMs,
                        deliveryTimeoutMs,
                        partitionerConfig,
                        metrics,
                        PRODUCER_METRIC_GROUP_NAME,
                        time,
                        transactionManager,
                        new BufferPool(this.totalMemorySize, ChunkedRecordAccumulator.CHUNK_SIZE, metrics, time, PRODUCER_METRIC_GROUP_NAME, BufferPool.AllocationMode.INCREMENTAL));

View on GitHub (pinned to c31c9215e1)

Solutions

Set compression.type=none if incremental allocation is the priority (e.g. many partitions, memory-constrained producer).
Otherwise drop buffer.memory.allocation.strategy back to full (the default) to retain compression; the producer will over-reserve buffer memory but compress normally.
Lower batch.size below ChunkedRecordAccumulator.CHUNK_SIZE so incremental is ignored and full strategy with compression is used (note the producer already logs a warning in this case).
Track KAFKA-20579 and re-enable incremental+compression once shipped.

Example fix

# before
props.put(ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG,
         ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
new KafkaProducer<>(props); // -> ConfigException

# after (pick ONE)
# Option A — keep compression, use full allocator
props.remove(ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
# Option B — keep incremental allocator, disable compression
props.put(ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG,
         ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "none");

Defensive patterns

Strategy: validation

Validate before calling

// The incremental buffer-memory allocation strategy does not yet support
// compression (KAFKA-20579). Validate the combination BEFORE constructing the
// KafkaProducer, otherwise KafkaProducer throws ConfigException.
import org.apache.kafka.clients.producer.ProducerConfig;
import java.util.Map;

static void assertConfigCompatible(Map<String, Object> props) {
    String strategy = String.valueOf(props.getOrDefault(
            ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG, "full"))
            .toLowerCase(java.util.Locale.ROOT);
    String compression = String.valueOf(props.getOrDefault(
            ProducerConfig.COMPRESSION_TYPE_CONFIG, "none")).toLowerCase(java.util.Locale.ROOT);
    // Only the incremental path that actually activates (batch.size >= chunk size)
    // is restricted; guard conservatively for any incremental use.
    if ("incremental".equals(strategy) && !"none".equals(compression)) {
        throw new org.apache.kafka.common.config.ConfigException(
            "buffer.memory.allocation.strategy=incremental requires compression.type=none");
    }
}

// usage:
assertConfigCompatible(props);
new KafkaProducer<>(props, keySer, valSer);

Try / catch

try {
    producer = new KafkaProducer<>(props);
} catch (org.apache.kafka.common.config.ConfigException e) {
    if (e.getMessage().contains("does not support compression")) {
        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "none");
        producer = new KafkaProducer<>(props);   // retry uncompressed
    } else { throw e; }
}

Prevention

If you need compression, do not set buffer.memory.allocation.strategy=incremental.
Keep producer configuration in one place and unit-test illegal combinations at startup.
Pin compression.type explicitly (default is none) so an inherited prop cannot surprise you.
Track KAFKA-20579 — once chunked compression ships, this constraint is lifted.

When it happens

Trigger: Producer config containing both buffer.memory.allocation.strategy=incremental (with batch.size at or above the chunk size threshold) and compression.type in {gzip, snappy, lz4, zstd}. Setting compression.type=none is the only way to keep incremental allocation.

Common situations: Adopting the new incremental allocator to cut buffer-pool memory while forgetting that production traffic relies on zstd/snappy compression; copy-pasted producer properties from a service that used the default (full) strategy into one that flips to incremental; bumping batch.size high enough to cross the CHUNK_SIZE threshold and suddenly activating the chunked path.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/c8b82207a0fe5f2f.json. Report an issue: GitHub.