{"id":"c8b82207a0fe5f2f","repo":"apache/kafka","slug":"the-producerconfig-buffer-memory-allocation-stra","errorCode":null,"errorMessage":"The ${ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL} ${ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG} does not support compression yet. ${ProducerConfig.COMPRESSION_TYPE_CONFIG} must be set to none.","messagePattern":"The (.+?) (.+?) does not support compression yet\\. (.+?) must be set to none\\.","errorType":"exception","errorClass":"ConfigException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java","lineNumber":477,"sourceCode":"            int batchSize = Math.max(1, config.getInt(ProducerConfig.BATCH_SIZE_CONFIG));\n            String allocationStrategy = config.getString(ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG)\n                    .toLowerCase(Locale.ROOT);\n            boolean incremental = ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL.equals(allocationStrategy);\n            // Use the chunked path only when a batch is at least one full chunk\n            // (batch.size >= CHUNK_SIZE). Below that, a batch can't fill even one chunk, so chunking\n            // would over-reserve and the producer falls back to the full strategy instead.\n            boolean useIncremental = incremental && batchSize >= ChunkedRecordAccumulator.CHUNK_SIZE;\n            if (incremental && !useIncremental) {\n                log.warn(\"Ignoring {}={} and falling back to {}: {} is {} bytes, below the {} byte chunk size, \" +\n                                \"so a batch cannot fill a single chunk.\",\n                        ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG,\n                        ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL,\n                        ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_FULL,\n                        ProducerConfig.BATCH_SIZE_CONFIG, batchSize, ChunkedRecordAccumulator.CHUNK_SIZE);\n            }\n            // The chunked path does not support compression yet (TODO: KAFKA-20579)\n            if (useIncremental && compression.type() != CompressionType.NONE) {\n                throw new ConfigException(\"The \" + ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL\n                        + \" \" + ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG\n                        + \" does not support compression yet. \" + ProducerConfig.COMPRESSION_TYPE_CONFIG\n                        + \" must be set to none.\");\n            }\n            if (useIncremental) {\n                this.accumulator = new ChunkedRecordAccumulator(logContext,\n                        batchSize,\n                        compression,\n                        lingerMs(config),\n                        retryBackoffMs,\n                        retryBackoffMaxMs,\n                        deliveryTimeoutMs,\n                        partitionerConfig,\n                        metrics,\n                        PRODUCER_METRIC_GROUP_NAME,\n                        time,\n                        transactionManager,\n                        new BufferPool(this.totalMemorySize, ChunkedRecordAccumulator.CHUNK_SIZE, metrics, time, PRODUCER_METRIC_GROUP_NAME, BufferPool.AllocationMode.INCREMENTAL));","sourceCodeStart":459,"sourceCodeEnd":495,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L459-L495","documentation":"Thrown during KafkaProducer construction when the chunked/incremental buffer allocation path is selected (buffer.memory.allocation.strategy=incremental AND batch.size >= ChunkedRecordAccumulator.CHUNK_SIZE) but compression is anything other than NONE. The chunked accumulator does not yet implement compression (tracked by KAFKA-20579), so the two together are an unsupported combination. The check is performed after the 'useIncremental' decision logic and aborts producer startup with a ConfigException.","triggerScenarios":"Producer config containing both buffer.memory.allocation.strategy=incremental (with batch.size at or above the chunk size threshold) and compression.type in {gzip, snappy, lz4, zstd}. Setting compression.type=none is the only way to keep incremental allocation.","commonSituations":"Adopting the new incremental allocator to cut buffer-pool memory while forgetting that production traffic relies on zstd/snappy compression; copy-pasted producer properties from a service that used the default (full) strategy into one that flips to incremental; bumping batch.size high enough to cross the CHUNK_SIZE threshold and suddenly activating the chunked path.","solutions":["Set compression.type=none if incremental allocation is the priority (e.g. many partitions, memory-constrained producer).","Otherwise drop buffer.memory.allocation.strategy back to full (the default) to retain compression; the producer will over-reserve buffer memory but compress normally.","Lower batch.size below ChunkedRecordAccumulator.CHUNK_SIZE so incremental is ignored and full strategy with compression is used (note the producer already logs a warning in this case).","Track KAFKA-20579 and re-enable incremental+compression once shipped."],"exampleFix":"# before\nprops.put(ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG,\n         ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL);\nprops.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, \"zstd\");\nnew KafkaProducer<>(props); // -> ConfigException\n\n# after (pick ONE)\n# Option A — keep compression, use full allocator\nprops.remove(ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG);\nprops.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, \"zstd\");\n# Option B — keep incremental allocator, disable compression\nprops.put(ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG,\n         ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCREMENTAL);\nprops.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, \"none\");","handlingStrategy":"validation","validationCode":"// The incremental buffer-memory allocation strategy does not yet support\n// compression (KAFKA-20579). Validate the combination BEFORE constructing the\n// KafkaProducer, otherwise KafkaProducer throws ConfigException.\nimport org.apache.kafka.clients.producer.ProducerConfig;\nimport java.util.Map;\n\nstatic void assertConfigCompatible(Map<String, Object> props) {\n    String strategy = String.valueOf(props.getOrDefault(\n            ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_CONFIG, \"full\"))\n            .toLowerCase(java.util.Locale.ROOT);\n    String compression = String.valueOf(props.getOrDefault(\n            ProducerConfig.COMPRESSION_TYPE_CONFIG, \"none\")).toLowerCase(java.util.Locale.ROOT);\n    // Only the incremental path that actually activates (batch.size >= chunk size)\n    // is restricted; guard conservatively for any incremental use.\n    if (\"incremental\".equals(strategy) && !\"none\".equals(compression)) {\n        throw new org.apache.kafka.common.config.ConfigException(\n            \"buffer.memory.allocation.strategy=incremental requires compression.type=none\");\n    }\n}\n\n// usage:\nassertConfigCompatible(props);\nnew KafkaProducer<>(props, keySer, valSer);","typeGuard":null,"tryCatchPattern":"try {\n    producer = new KafkaProducer<>(props);\n} catch (org.apache.kafka.common.config.ConfigException e) {\n    if (e.getMessage().contains(\"does not support compression\")) {\n        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, \"none\");\n        producer = new KafkaProducer<>(props);   // retry uncompressed\n    } else { throw e; }\n}","preventionTips":["If you need compression, do not set buffer.memory.allocation.strategy=incremental.","Keep producer configuration in one place and unit-test illegal combinations at startup.","Pin compression.type explicitly (default is none) so an inherited prop cannot surprise you.","Track KAFKA-20579 — once chunked compression ships, this constraint is lifted."],"tags":["producer","config","compression","buffer-pool","allocation-strategy"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}