apache/kafka · error · ConfigException

To use the idempotent producer, max.in.flight.requests.per.c

Error message

To use the idempotent producer, max.in.flight.requests.per.connection must be set to at most 5. Current value is {inFlightConnection}.

What it means

Thrown by ProducerConfig.postProcessAndValidateIdempotenceConfigs when idempotence is enabled and max.in.flight.requests.per.connection exceeds 5. The broker deduplicates a producer's writes per partition using a monotonically increasing sequence number, and with more than 5 unacknowledged in-flight requests a retry could reorder batches beyond what the broker can de-dupe, breaking the idempotent guarantee. Unlike retries/acks, this is always a hard failure regardless of whether idempotence was explicitly set.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java:674

                    throw new ConfigException("Must set " + RETRIES_CONFIG + " to non-zero when using the idempotent producer.");
                }
                log.info("Idempotence will be disabled because {} is set to 0.", RETRIES_CONFIG);
                shouldDisableIdempotence = true;
            }

            final short acks = Short.parseShort(acksStr);
            if (acks != (short) -1) {
                if (userConfiguredIdempotence) {
                    throw new ConfigException("Must set " + ACKS_CONFIG + " to all in order to use the idempotent " +
                        "producer. Otherwise we cannot guarantee idempotence.");
                }
                log.info("Idempotence will be disabled because {} is set to {}, not set to 'all'.", ACKS_CONFIG, acks);
                shouldDisableIdempotence = true;
            }

            final int inFlightConnection = this.getInt(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION);
            if (MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE < inFlightConnection) {
                throw new ConfigException("To use the idempotent producer, " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION +
                                          " must be set to at most 5. Current value is " + inFlightConnection + ".");
            }
        }

        if (shouldDisableIdempotence) {
            configs.put(ENABLE_IDEMPOTENCE_CONFIG, false);
            idempotenceEnabled = false;
        }

        // validate `transaction.id` after validating idempotence dependant configs because `enable.idempotence` config might be overridden
        boolean userConfiguredTransactions = originalConfigs.containsKey(TRANSACTIONAL_ID_CONFIG);
        if (!idempotenceEnabled && userConfiguredTransactions) {
            throw new ConfigException("Cannot set a " + ProducerConfig.TRANSACTIONAL_ID_CONFIG + " without also enabling idempotence.");
        }

        // Validate that transaction.timeout.ms is not set when transaction.two.phase.commit.enable is true
        // In standard Kafka transactions, the broker enforces transaction.timeout.ms and aborts any
        // transaction that isn't completed in time. With two-phase commit (2PC), an external coordinator

View on GitHub (pinned to c31c9215e1)

Solutions

Lower `max.in.flight.requests.per.connection` to 5 or less (1-5 are the valid values for the idempotent producer).
If higher concurrency is required, disable idempotence and transactional.id — but be aware you lose exactly-once and may get duplicates on retry.
Audit existing producer property files and Spring/Kafka Streams configs that override this value when upgrading the client.

Example fix

// before
props.put("enable.idempotence", "true");
props.put("max.in.flight.requests.per.connection", "10");

// after
props.put("enable.idempotence", "true");
props.put("max.in.flight.requests.per.connection", "5");

Defensive patterns

Strategy: validation

Validate before calling

// Cap max.in.flight to <= 5 for idempotent producers, BEFORE constructing.
boolean idempotenceEnabled = Boolean.TRUE.equals(cfg.get(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG));
int inFlight = ((Number) cfg.getOrDefault(
    ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5)).intValue();
if (idempotenceEnabled && inFlight > 5) {
    throw new IllegalStateException(
        "max.in.flight.requests.per.connection must be <= 5 with idempotence");
}
// Or clamp it:
if (idempotenceEnabled && inFlight > 5)
    cfg.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);

Try / catch

try {
    producer = new KafkaProducer<>(cfg);
} catch (ConfigException e) {
    if (e.getMessage().contains("max.in.flight")) {
        cfg.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
        producer = new KafkaProducer<>(cfg);
    } else throw e;
}

Prevention

The idempotent cap is 5; leave the default (5) unless you have a specific reason to lower it.
Throughput tuning via high max.in.flight is incompatible with idempotence — tune batch.size/linger.ms instead.
Unit-test your config builder to assert in-flight <= 5 whenever idempotence is enabled.

When it happens

Trigger: Constructing a KafkaProducer with `enable.idempotence=true` (default or explicit) and `max.in.flight.requests.per.connection=6` or higher. The constant MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE is 5, checked at ProducerConfig.java:673.

Common situations: Older tutorials and legacy configs set max.in.flight to 10 or more for throughput on a non-idempotent producer; modern Kafka clients default enable.idempotence=true, turning that previously-fine value into a hard error on producer construction.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/907a4160706604aa.json. Report an issue: GitHub.