What causes "Operation timed out - received 0/%d responses." in apache/cassandra?

Retry the operation with backoff — Paxos locks are transient contention, not permanent failure

apache/cassandra · warning · WriteTimeoutException

WRITE_TIMEOUT

WRITE_TIMEOUT

Error message

Operation timed out - received 0/%d responses.

What it means

PaxosState.lock() attempts to acquire the per-key Paxos lock within the caller's deadline. If the lock cannot be acquired before the deadline expires, a TimeoutException (code WRITE_TIMEOUT) is thrown with a message reporting how many responses were received out of the required block-for (here 0). This surfaces that the coordinator could not even begin the consensus round in time.

Source

Thrown at src/java/org/apache/cassandra/service/paxos/PaxosState.java:429

    // don't increment the total count, as we are only using this for locking purposes when coordinating
    @VisibleForTesting
    public static PaxosOperationLock lock(DecoratedKey partitionKey, TableMetadata metadata, long deadline, ConsistencyLevel consistencyForConsensus, boolean isWrite) throws RequestTimeoutException
    {
        if (DISABLE_COORDINATOR_LOCKING)
            return PaxosOperationLock.noOp();

        PaxosState lock = ACTIVE.compute(new Key(partitionKey, metadata), (key, cur) -> {
            if (cur == null)
                cur = new PaxosState(key, RECENT.remove(key));
            ++cur.active;
            return cur;
        });

        try
        {
            if (!lock.lock(deadline))
                throw throwTimeout(metadata, consistencyForConsensus, isWrite);
            return lock;
        }
        catch (Throwable t)
        {
            lock.close();
            throw t;
        }
    }
    
    private static RequestTimeoutException throwTimeout(TableMetadata metadata, ConsistencyLevel consistencyForConsensus, boolean isWrite)
    {
        int blockFor = consistencyForConsensus.blockFor(Keyspace.open(metadata.keyspace).getReplicationStrategy());
        throw isWrite
                ? new WriteTimeoutException(WriteType.CAS, consistencyForConsensus, 0, blockFor)
                : new ReadTimeoutException(consistencyForConsensus, 0, blockFor, false);
    }

    private PaxosState maybeLoad()

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Retry the operation with backoff — Paxos locks are transient contention, not permanent failure
Reduce concurrent writers to the same partition key or shard the hot key
Increase the client write timeout / USING TIMEOUT for LWT-heavy workloads
Check replica health (GC pauses, dropped messages, network latency) with nodetool tpstats/nodetool netstats
Consider reducing CAS usage on hot keys or using a different data model

Example fix

// before
session.execute("UPDATE t SET v = 1 WHERE k = ? IF v = 0", key); // WRITE_TIMEOUT under contention
// after
for (int attempt = 0; attempt < 3; attempt++) {
    try {
        session.execute("UPDATE t SET v = 1 WHERE k = ? IF v = 0", key);
        break;
    } catch (WriteTimeoutException e) {
        Thread.sleep(50L * (attempt + 1));
    }
}

Defensive patterns

Strategy: retry

Try / catch

try {
    return paxosOperation();
} catch (WriteTimeoutException e) {
    if (e.blockFor() == 0 || e.receivedAcknowledgments() == 0)
        backoffAndRetry(e, attempt); // lock contention, safe to retry
    else
        throw e;
}

Prevention

Avoid hot keys with high CAS concurrency; shard or queue writes per key
Set generous timeouts for LWT statements (they need multiple round trips)
Monitor replica GC pauses and dropped mutations that delay lock grants
Implement bounded retry with jittered backoff for Paxos WRITE_TIMEOUT with 0 responses

When it happens

Trigger: Calling PaxosState.lock() under write request timeout pressure: concurrent Paxos transactions contending on the same partition key, or a node/network too slow to grant the lock before the deadline (e.g. a query with low USING TIMEOUT or an overloaded cluster).

Common situations: Hot-partition contention where many LWT (lightweight transaction) writes race on the same row; severely overloaded or GC-pausing replicas; client-side timeouts set too low for CAS operations; Repair/Paxos ballot storms.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/346ca0c0ec4f5d91. Report an issue: GitHub.