apache/cassandra · error · OverloadedException

Too many in flight hints

Error message

Too many in flight hints: %s destination: %s destination hints: %s

What it means

OverloadedException thrown by StorageProxy when the cluster-wide count of hints in flight exceeds maxHintsInProgress AND the specific destination already has hints pending. Cassandra deliberately rejects writes that would require hinting to an already-backlogged destination so that a few problem nodes do not stall writes to healthy ones.

Solutions

1. Restore/downplay the problematic destination: bring the down node back or drain its hint backlog (check `nodetool statusbackup`/hinted handoff metrics).
2. Raise max_hints_in_progress in cassandra.yaml (or via JMX StorageService) if the cluster can tolerate more concurrent hints.
3. Disable or tune hinted handoff (hinted_handoff_enabled, max_hint_window_in_ms) if hints are not required for your consistency strategy.
4. Catch OverloadedException on the client and retry with backoff; the write is intentionally rejected, not lost by design.

Example fix

// before (client sends write at full speed during node flap)
session.execute(writeStmt);
// after
catch (OverloadedException e) {
    Thread.sleep(backoffMs);
    session.execute(writeStmt); // retry after backoff
}

long inFlight = ((Number) jmxConn.getAttribute(hintMetrics, "TotalHintsInProgress")).longValue();
if (inFlight > maxHintsInProgress) backoffBeforeWrite(destination);

catch (OverloadedException e) {
    sleep(exponentialBackoff(attempt));
    retryWrite();
}

When it happens

Trigger: A write to a replica that is down/unreachable requires a hint while StorageMetrics.totalHintsInProgress > maxHintsInProgress and getHintsInProgressFor(destination) > 0. Typical calls: any MutationToken write via StorageProxy.mutateWithViewBuilder/mutate/applyCounterMutation with hint window enabled and a flapped or slow node.

Common situations: A node is down longer than max_hint_window but still hinted to, hint queues saturated by a partitioned node, maxHintsInProgress (default 128) too low for write-heavy clusters, storage on a destination node too slow to drain hints.

Related errors

• A maximum number of tokens per node is supported
• A repair_session_space of
• A repair_session_space of
• A storage-attached index cannot be created over multiple…
• A tombstone should not have a value

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/341505021a0ea450. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/service/StorageProxy.java:1945

        if (dcGroups != null)
        {
            // for each datacenter, send the message to one node to relay the write to other replicas
            for (Collection<Replica> dcTargets : dcGroups.values())
                sendMessagesToNonlocalDC(message, EndpointsForToken.copyOf(mutation.key().getToken(), dcTargets), responseHandler);
        }
    }

    private static void checkHintOverload(Replica destination)
    {
        // avoid OOMing due to excess hints.  we need to do this check even for "live" nodes, since we can
        // still generate hints for those if it's overloaded or simply dead but not yet known-to-be-dead.
        // The idea is that if we have over maxHintsInProgress hints in flight, this is probably due to
        // a small number of nodes causing problems, so we should avoid shutting down writes completely to
        // healthy nodes.  Any node with no hintsInProgress is considered healthy.
        if (StorageMetrics.totalHintsInProgress.getCount() > maxHintsInProgress
                && (getHintsInProgressFor(destination.endpoint()).get() > 0 && shouldHint(destination)))
        {
            throw new OverloadedException("Too many in flight hints: " + StorageMetrics.totalHintsInProgress.getCount() +
                                          " destination: " + destination +
                                          " destination hints: " + getHintsInProgressFor(destination.endpoint()).get());
        }
    }

    /*
     * Send the message to the first replica of targets, and have it forward the message to others in its DC
     */
    private static void sendMessagesToNonlocalDC(Message<? extends IMutation> message,
                                                 EndpointsForToken targets,
                                                 AbstractWriteResponseHandler<IMutation> handler)
    {
        final Replica target;

        if (targets.size() > 1)
        {
            target = pickReplica(targets);
            EndpointsForToken forwardToReplicas = targets.filter(r -> r != target, targets.size());

View on GitHub (pinned to 88fd0f6a0e)