{"id":"907a4160706604aa","repo":"apache/kafka","slug":"to-use-the-idempotent-producer-max-in-flight-requ","errorCode":null,"errorMessage":"To use the idempotent producer, max.in.flight.requests.per.connection must be set to at most 5. Current value is {inFlightConnection}.","messagePattern":"To use the idempotent producer, max\\.in\\.flight\\.requests\\.per\\.connection must be set to at most 5\\. Current value is (.+?)\\.","errorType":"validation","errorClass":"ConfigException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java","lineNumber":674,"sourceCode":"                    throw new ConfigException(\"Must set \" + RETRIES_CONFIG + \" to non-zero when using the idempotent producer.\");\n                }\n                log.info(\"Idempotence will be disabled because {} is set to 0.\", RETRIES_CONFIG);\n                shouldDisableIdempotence = true;\n            }\n\n            final short acks = Short.parseShort(acksStr);\n            if (acks != (short) -1) {\n                if (userConfiguredIdempotence) {\n                    throw new ConfigException(\"Must set \" + ACKS_CONFIG + \" to all in order to use the idempotent \" +\n                        \"producer. Otherwise we cannot guarantee idempotence.\");\n                }\n                log.info(\"Idempotence will be disabled because {} is set to {}, not set to 'all'.\", ACKS_CONFIG, acks);\n                shouldDisableIdempotence = true;\n            }\n\n            final int inFlightConnection = this.getInt(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION);\n            if (MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE < inFlightConnection) {\n                throw new ConfigException(\"To use the idempotent producer, \" + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION +\n                                          \" must be set to at most 5. Current value is \" + inFlightConnection + \".\");\n            }\n        }\n\n        if (shouldDisableIdempotence) {\n            configs.put(ENABLE_IDEMPOTENCE_CONFIG, false);\n            idempotenceEnabled = false;\n        }\n\n        // validate `transaction.id` after validating idempotence dependant configs because `enable.idempotence` config might be overridden\n        boolean userConfiguredTransactions = originalConfigs.containsKey(TRANSACTIONAL_ID_CONFIG);\n        if (!idempotenceEnabled && userConfiguredTransactions) {\n            throw new ConfigException(\"Cannot set a \" + ProducerConfig.TRANSACTIONAL_ID_CONFIG + \" without also enabling idempotence.\");\n        }\n\n        // Validate that transaction.timeout.ms is not set when transaction.two.phase.commit.enable is true\n        // In standard Kafka transactions, the broker enforces transaction.timeout.ms and aborts any\n        // transaction that isn't completed in time. With two-phase commit (2PC), an external coordinator","sourceCodeStart":656,"sourceCodeEnd":692,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java#L656-L692","documentation":"Thrown by ProducerConfig.postProcessAndValidateIdempotenceConfigs when idempotence is enabled and max.in.flight.requests.per.connection exceeds 5. The broker deduplicates a producer's writes per partition using a monotonically increasing sequence number, and with more than 5 unacknowledged in-flight requests a retry could reorder batches beyond what the broker can de-dupe, breaking the idempotent guarantee. Unlike retries/acks, this is always a hard failure regardless of whether idempotence was explicitly set.","triggerScenarios":"Constructing a KafkaProducer with `enable.idempotence=true` (default or explicit) and `max.in.flight.requests.per.connection=6` or higher. The constant MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE is 5, checked at ProducerConfig.java:673.","commonSituations":"Older tutorials and legacy configs set max.in.flight to 10 or more for throughput on a non-idempotent producer; modern Kafka clients default enable.idempotence=true, turning that previously-fine value into a hard error on producer construction.","solutions":["Lower `max.in.flight.requests.per.connection` to 5 or less (1-5 are the valid values for the idempotent producer).","If higher concurrency is required, disable idempotence and transactional.id — but be aware you lose exactly-once and may get duplicates on retry.","Audit existing producer property files and Spring/Kafka Streams configs that override this value when upgrading the client."],"exampleFix":"// before\nprops.put(\"enable.idempotence\", \"true\");\nprops.put(\"max.in.flight.requests.per.connection\", \"10\");\n\n// after\nprops.put(\"enable.idempotence\", \"true\");\nprops.put(\"max.in.flight.requests.per.connection\", \"5\");","handlingStrategy":"validation","validationCode":"// Cap max.in.flight to <= 5 for idempotent producers, BEFORE constructing.\nboolean idempotenceEnabled = Boolean.TRUE.equals(cfg.get(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG));\nint inFlight = ((Number) cfg.getOrDefault(\n    ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5)).intValue();\nif (idempotenceEnabled && inFlight > 5) {\n    throw new IllegalStateException(\n        \"max.in.flight.requests.per.connection must be <= 5 with idempotence\");\n}\n// Or clamp it:\nif (idempotenceEnabled && inFlight > 5)\n    cfg.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);","typeGuard":null,"tryCatchPattern":"try {\n    producer = new KafkaProducer<>(cfg);\n} catch (ConfigException e) {\n    if (e.getMessage().contains(\"max.in.flight\")) {\n        cfg.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);\n        producer = new KafkaProducer<>(cfg);\n    } else throw e;\n}","preventionTips":["The idempotent cap is 5; leave the default (5) unless you have a specific reason to lower it.","Throughput tuning via high max.in.flight is incompatible with idempotence — tune batch.size/linger.ms instead.","Unit-test your config builder to assert in-flight <= 5 whenever idempotence is enabled."],"tags":["kafka","producer","configuration","idempotence","throughput"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}