apache/cassandra · error · ReadTimeoutException

ReadTimeoutException

Error message

ReadTimeoutException

What it means

BlockingReadRepair.awaitWrites throws ReadTimeoutException when the read-repair write phase triggered by a digest mismatch does not complete in time. It reports that all data/digest responses arrived but the repair writes timed out, surfaced with received = min(blockFor - waitingOn, blockFor - 1) so the client sees a read timeout attributed to the repair.

Solutions

1. Increase write_request_timeout_in_ms / read_request_timeout_in_ms if repairs legitimately need longer
2. Run nodetool repair to bring replicas back in sync so reads stop triggering large blocking repairs
3. Check replica health (compactions backlog, dropped messages, GC pauses) that makes repair writes slow
4. Reduce inconsistency window: fix hinted-handoff backlogs and monitor digest-mismatch rates

Example fix

// before
cassandra.yaml: write_request_timeout_in_ms: 2000
// after
cassandra.yaml: write_request_timeout_in_ms: 5000

// Before reads: check replica sync state
// shell: nodetool repair -pr (scheduled) and monitor 'Read repair' metrics; skip strict-CL reads if repair backlog is large

try {
  return session.execute(query);
} catch (e) {
  if (e instanceof ReadTimeoutException) {
    // repair writes timed out; retry after backoff, then escalate to nodetool repair
    return retryWithBackoff(query, 2).catch(() => scheduleRepair(keyspace, table));
  }
  throw e;
}

When it happens

Trigger: A read with CL requiring multiple replicas finds a digest mismatch; the blocking repair writes mutations to out-of-date replicas and awaits their acknowledgments; those writes hit the write timeout (slow replica, overload, tombstone-heavy repair).

Common situations: Replicas out of sync for a long time (a node down then returning) so repairs move a lot of data; TombstoneOverwhelming during repair writes; disk contention making the repair writes exceed write_request_timeout.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• ReadTimeoutException
• ReadTimeoutException
• ReadTimeoutException
• Attempting to load denylist and not enough nodes are…
• Auth check after connection closed

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/924bf2f3b9852dcc. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/service/reads/repair/BlockingReadRepair.java:192

            if (!repair.awaitRepairsUntil(deadline, NANOSECONDS))
            {
                timedOut = repair;
                break;
            }
            repairPlan = repair.repairPlan();
        }
        if (timedOut != null)
        {
            // We got all responses, but timed out while repairing;
            // pick one of the repairs to throw, as this is better than completely manufacturing the error message
            int blockFor = timedOut.blockFor();
            int received = Math.min(blockFor - timedOut.waitingOn(), blockFor - 1);
            if (Tracing.isTracing())
                Tracing.trace("Timed out while read-repairing after receiving all {} data and digest responses", blockFor);
            else
                logger.debug("Timeout while read-repairing after receiving all {} data and digest responses", blockFor);

            throw new ReadTimeoutException(replicaPlan().consistencyLevel(), received, blockFor, true);
        }

        if (repairs.isEmpty() || repairPlan.stillAppliesTo(ClusterMetadata.current()))
            return;
    }

    @Override
    public void repairPartition(DecoratedKey dk, Map<Replica, Mutation> mutations, ReplicaPlan.ForWrite writePlan, ReadRepairSource rrSource)
    {
        // non-Accord reads only ever touch one table and key so all mutations need to be applied either transactionally
        // or non-transactionally (not a mix). There is no retry loop here because read repair is relatively rare so it racing
        // with changes to migrating ranges should also be pretty rare so it isn't worth the added complexity. If you were
        // to add a retry loop you would need to be careful to correctly set/unset allowPotentialTransactionConflicts in the mutations
        // since that is set if it is routed to Accord
        //
        // If this is an Accord transaction that is in interoperability mode and executing a read repair
        // then we take the non-transactional path and the mutations are intercepted in ReadCoordinator.sendRepairMutation
        // which will ensure the repair mutation runs in the command store thread after preceding transactions are done

View on GitHub (pinned to 88fd0f6a0e)