apache/kafka · error · IllegalStateException

Producer with transactionalId '{transactionalId}' and {produ

Error message

Producer with transactionalId '{transactionalId}' and {producerIdAndEpoch} cannot execute transactional method because of previous invalid state transition attempt

What it means

Thrown by TransactionManager.maybeFailWithError when the cached lastError is an IllegalStateException, i.e. an earlier call caused an invalid state-machine transition and poisoned the producer into State.FATAL_ERROR (see shouldPoisonStateOnInvalidTransition). Any later transactional method invokes maybeFailOnError and rewraps the original failure with this message, identifying the transactionalId and producerId/epoch. The producer is permanently unusable because the state machine is in FATAL_ERROR and cannot recover without a new producer instance.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:1196

    }

    private void maybeFailWithError() {
        if (!hasError()) {
            return;
        }
        // for ProducerFencedException, do not wrap it as a KafkaException
        // but create a new instance without the call trace since it was not thrown because of the current call
        if (lastError instanceof ProducerFencedException) {
            throw new ProducerFencedException("Producer with transactionalId '" + transactionalId
                    + "' and " + producerIdAndEpoch + " has been fenced by another producer " +
                    "with the same transactionalId");
        }
        if (lastError instanceof InvalidProducerEpochException) {
            throw new InvalidProducerEpochException("Producer with transactionalId '" + transactionalId
                    + "' and " + producerIdAndEpoch + " attempted to produce with an old epoch");
        }
        if (lastError instanceof IllegalStateException) {
            throw new IllegalStateException("Producer with transactionalId '" + transactionalId
                    + "' and " + producerIdAndEpoch + " cannot execute transactional method because of previous invalid state transition attempt", lastError);
        }
        throw new KafkaException("Cannot execute transactional method because we are in an error state", lastError);
    }

    private boolean maybeTerminateRequestWithError(TxnRequestHandler requestHandler) {
        if (hasError()) {
            if (hasAbortableError() && requestHandler instanceof FindCoordinatorHandler)
                // No harm letting the FindCoordinator request go through if we're expecting to abort
                return false;

            requestHandler.fail(lastError);
            return true;
        }
        return false;
    }

    private void enqueueRequest(TxnRequestHandler requestHandler) {

View on GitHub (pinned to c31c9215e1)

Solutions

Audit the call sequence around the first failure: ensure the pattern is strictly initTransactions -> beginTransaction -> send/sendOffsetsToTransaction -> commitTransaction (or abortTransaction) with no out-of-order or duplicate transitions.
Once the producer is poisoned to FATAL_ERROR it cannot be recovered — close() it and create a new KafkaProducer with initTransactions().
Make the producer access single-threaded or serialize transactional calls through the Sender thread; KafkaProducer is not safe for concurrent transactional calls.
Add unit/integration tests covering the exact transaction lifecycle used by your code path to catch sequencing bugs at build time.

Example fix

// before
producer.initTransactions();
producer.beginTransaction();
producer.commitTransaction();
producer.commitTransaction();  // illegal transition -> FATAL_ERROR, next call throws 291

// after
producer.initTransactions();
producer.beginTransaction();
producer.send(record);
producer.commitTransaction();
// start a new transaction explicitly before committing again
producer.beginTransaction();
producer.send(record2);
producer.commitTransaction();

Defensive patterns

Strategy: try-catch

Validate before calling

// No public API exposes the internal state machine. The only safe pre-check is to
// ensure the producer is not in an error state by never ignoring prior exceptions.
// Maintain your own flag and set it whenever any transactional call throws:
class TxGuard {
    private volatile boolean poisoned = false;
    void markFailed() { poisoned = true; }
    boolean isUsable(KafkaProducer<?,?> p) { return !poisoned; }
}

Type guard

public static boolean isProducerInCleanState(KafkaProducer<?,?> p) {
    // Reflectively read hasError(); not part of the public contract, use cautiously.
    try {
        java.lang.reflect.Field tm = p.getClass().getDeclaredField("transactionManager");
        tm.setAccessible(true);
        Object manager = tm.get(p);
        java.lang.reflect.Method hasError = manager.getClass().getDeclaredMethod("hasError");
        hasError.setAccessible(true);
        return !((Boolean) hasError.invoke(manager));
    } catch (Exception ex) { return true; }
}

Try / catch

try {
    producer.commitTransaction();
} catch (IllegalStateException e) {
    if (e.getMessage() != null && e.getMessage().contains("invalid state transition")) {
        // Producer was driven through an illegal sequence (e.g. commit without begin).
        // The producer is poisoned; abandon and recreate it.
        producer.close(Duration.ZERO);
        throw new IllegalStateException("Producer mis-sequenced and is now poisoned; recreate it", e);
    }
    throw e;
}

Prevention

Always follow the exact lifecycle: initTransactions() once, then beginTransaction() before every send/commit/abort, and never call commit/abort twice in a row.
Treat ANY exception during a transactional call as fatal for that producer instance: abort, close, and instantiate a new one rather than retrying on the same object.
Avoid sending offset commits (sendOffsetsToTransaction) before beginTransaction(); verify current state in your own wrapper before each transition.

When it happens

Trigger: Calling beginTransaction/commitTransaction/abortTransaction/sendOffsetsToTransaction after a prior call to the same or another transactional method performed an illegal transition (e.g. beginTransaction called twice without commit, commitTransaction when not in a transaction, or send before beginTransaction triggered transition to an error state). The first illegal transition set lastError to IllegalStateException and currentState to FATAL_ERROR; this exception surfaces on the next transactional call.

Common situations: Application logic bug in transaction sequencing (calling commit twice, aborting after commit, forgetting beginTransaction); library/framework wrapping KafkaProducer that retries the same operation after a partial failure without resetting state; migration from non-EOS to EOS producer without updating call ordering; concurrent threads invoking transactional methods on a shared non-thread-safe producer.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/33218d2b8e7227d2.json. Report an issue: GitHub.