apache/kafka · error · ConfigException

Cannot set a transactional.id without also enabling idempote

Error message

Cannot set a transactional.id without also enabling idempotence.

What it means

Thrown by ProducerConfig.postProcessAndValidateIdempotenceConfigs when the user sets transactional.id but idempotence is not enabled after post-processing. Kafka's transactional producer builds on the idempotent producer (producerId/epoch + sequence numbers), so a transactional.id without idempotence is meaningless; this guard runs after the retries/acks auto-disable logic so it also catches the case where idempotence was implicitly turned off.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java:687

                shouldDisableIdempotence = true;
            }

            final int inFlightConnection = this.getInt(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION);
            if (MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE < inFlightConnection) {
                throw new ConfigException("To use the idempotent producer, " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION +
                                          " must be set to at most 5. Current value is " + inFlightConnection + ".");
            }
        }

        if (shouldDisableIdempotence) {
            configs.put(ENABLE_IDEMPOTENCE_CONFIG, false);
            idempotenceEnabled = false;
        }

        // validate `transaction.id` after validating idempotence dependant configs because `enable.idempotence` config might be overridden
        boolean userConfiguredTransactions = originalConfigs.containsKey(TRANSACTIONAL_ID_CONFIG);
        if (!idempotenceEnabled && userConfiguredTransactions) {
            throw new ConfigException("Cannot set a " + ProducerConfig.TRANSACTIONAL_ID_CONFIG + " without also enabling idempotence.");
        }

        // Validate that transaction.timeout.ms is not set when transaction.two.phase.commit.enable is true
        // In standard Kafka transactions, the broker enforces transaction.timeout.ms and aborts any
        // transaction that isn't completed in time. With two-phase commit (2PC), an external coordinator
        // decides when to finalize, so broker-side timeouts don't apply. Disallow using both.
        boolean enable2PC = this.getBoolean(TRANSACTION_TWO_PHASE_COMMIT_ENABLE_CONFIG);
        boolean userConfiguredTransactionTimeout = originalConfigs.containsKey(TRANSACTION_TIMEOUT_CONFIG);
        if (enable2PC && userConfiguredTransactionTimeout) {
            throw new ConfigException(
                "Cannot set " + ProducerConfig.TRANSACTION_TIMEOUT_CONFIG +
                " when " + ProducerConfig.TRANSACTION_TWO_PHASE_COMMIT_ENABLE_CONFIG +
                " is set to true. Transactions will not expire with two-phase commit enabled."
            );
        }
    }

    private static String parseAcks(String acksString) {

View on GitHub (pinned to c31c9215e1)

Solutions

Set `enable.idempotence=true` (or remove an `enable.idempotence=false` override) so it stays on alongside transactional.id.
Ensure `retries` is non-zero and `acks=all` so idempotence is not silently auto-disabled before the transactional.id check runs.
If transactions are not actually needed, remove the `transactional.id` config.

Example fix

// before
props.put("transactional.id", "my-tx-producer");
props.put("enable.idempotence", "false");

// after
props.put("transactional.id", "my-tx-producer");
props.put("enable.idempotence", "true");
props.put("acks", "all");
props.put("retries", String.valueOf(Integer.MAX_VALUE));

Defensive patterns

Strategy: validation

Validate before calling

// transactional.id requires idempotence — check BEFORE constructing.
boolean hasTxnId = cfg.containsKey(ProducerConfig.TRANSACTIONAL_ID_CONFIG)
    && cfg.get(ProducerConfig.TRANSACTIONAL_ID_CONFIG) != null;
boolean idempotenceEnabled = Boolean.TRUE.equals(
    cfg.getOrDefault(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, false));
if (hasTxnId && !idempotenceEnabled) {
    throw new IllegalStateException(
        "transactional.id requires enable.idempotence=true");
}
// Simplest: enable idempotence implicitly whenever a transactional.id is set.
if (hasTxnId) cfg.putIfAbsent(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);

Try / catch

try {
    producer = new KafkaProducer<>(cfg);
} catch (ConfigException e) {
    if (e.getMessage().contains("transactional.id")) {
        cfg.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
        producer = new KafkaProducer<>(cfg);
    } else throw e;
}

Prevention

Transactional producers always imply idempotence — couple them in your config builder.
Guard against operators clearing enable.idempotence while leaving transactional.id set.
If you only want idempotence (not transactions), simply don't set transactional.id.

When it happens

Trigger: User provides `transactional.id=<some-id>` together with any combination that leaves idempotence off: explicitly `enable.idempotence=false`, or implicitly via `retries=0` / `acks!=all` combined with non-explicit idempotence (which silently disables it). The check at ProducerConfig.java:686 uses originals().containsKey on transactional.id to require the user to have set it.

Common situations: Setting transactional.id for EOS/transactional sends while inheriting a config that disabled idempotence, or explicitly setting enable.idempotence=false not realizing that transactional.id requires it.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/6c1a0f02080c9653.json. Report an issue: GitHub.