{"record":{"id":"341505021a0ea450","repo":"apache/cassandra","slug":"too-many-in-flight-hints-s-destination-s-desti","errorCode":null,"errorMessage":"Too many in flight hints: %s destination: %s destination hints: %s","messagePattern":"Too many in flight hints: (.+?) destination: (.+?) destination hints: (.+?)","errorType":"exception","errorClass":"OverloadedException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/service/StorageProxy.java","lineNumber":1945,"sourceCode":"        if (dcGroups != null)\n        {\n            // for each datacenter, send the message to one node to relay the write to other replicas\n            for (Collection<Replica> dcTargets : dcGroups.values())\n                sendMessagesToNonlocalDC(message, EndpointsForToken.copyOf(mutation.key().getToken(), dcTargets), responseHandler);\n        }\n    }\n\n    private static void checkHintOverload(Replica destination)\n    {\n        // avoid OOMing due to excess hints.  we need to do this check even for \"live\" nodes, since we can\n        // still generate hints for those if it's overloaded or simply dead but not yet known-to-be-dead.\n        // The idea is that if we have over maxHintsInProgress hints in flight, this is probably due to\n        // a small number of nodes causing problems, so we should avoid shutting down writes completely to\n        // healthy nodes.  Any node with no hintsInProgress is considered healthy.\n        if (StorageMetrics.totalHintsInProgress.getCount() > maxHintsInProgress\n                && (getHintsInProgressFor(destination.endpoint()).get() > 0 && shouldHint(destination)))\n        {\n            throw new OverloadedException(\"Too many in flight hints: \" + StorageMetrics.totalHintsInProgress.getCount() +\n                                          \" destination: \" + destination +\n                                          \" destination hints: \" + getHintsInProgressFor(destination.endpoint()).get());\n        }\n    }\n\n    /*\n     * Send the message to the first replica of targets, and have it forward the message to others in its DC\n     */\n    private static void sendMessagesToNonlocalDC(Message<? extends IMutation> message,\n                                                 EndpointsForToken targets,\n                                                 AbstractWriteResponseHandler<IMutation> handler)\n    {\n        final Replica target;\n\n        if (targets.size() > 1)\n        {\n            target = pickReplica(targets);\n            EndpointsForToken forwardToReplicas = targets.filter(r -> r != target, targets.size());","sourceCodeStart":1927,"sourceCodeEnd":1963,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/StorageProxy.java#L1927-L1963","documentation":"OverloadedException thrown by StorageProxy when the cluster-wide count of hints in flight exceeds maxHintsInProgress AND the specific destination already has hints pending. Cassandra deliberately rejects writes that would require hinting to an already-backlogged destination so that a few problem nodes do not stall writes to healthy ones.","triggerScenarios":"A write to a replica that is down/unreachable requires a hint while StorageMetrics.totalHintsInProgress > maxHintsInProgress and getHintsInProgressFor(destination) > 0. Typical calls: any MutationToken write via StorageProxy.mutateWithViewBuilder/mutate/applyCounterMutation with hint window enabled and a flapped or slow node.","commonSituations":"A node is down longer than max_hint_window but still hinted to, hint queues saturated by a partitioned node, maxHintsInProgress (default 128) too low for write-heavy clusters, storage on a destination node too slow to drain hints.","solutions":["Restore/downplay the problematic destination: bring the down node back or drain its hint backlog (check `nodetool statusbackup`/hinted handoff metrics).","Raise max_hints_in_progress in cassandra.yaml (or via JMX StorageService) if the cluster can tolerate more concurrent hints.","Disable or tune hinted handoff (hinted_handoff_enabled, max_hint_window_in_ms) if hints are not required for your consistency strategy.","Catch OverloadedException on the client and retry with backoff; the write is intentionally rejected, not lost by design."],"exampleFix":"// before (client sends write at full speed during node flap)\nsession.execute(writeStmt);\n// after\ncatch (OverloadedException e) {\n    Thread.sleep(backoffMs);\n    session.execute(writeStmt); // retry after backoff\n}","handlingStrategy":"retry","validationCode":"long inFlight = ((Number) jmxConn.getAttribute(hintMetrics, \"TotalHintsInProgress\")).longValue();\nif (inFlight > maxHintsInProgress) backoffBeforeWrite(destination);","typeGuard":null,"tryCatchPattern":"catch (OverloadedException e) {\n    sleep(exponentialBackoff(attempt));\n    retryWrite();\n}","preventionTips":["Monitor StorageMetrics.totalHintsInProgress and alert near maxHintsInProgress.","Keep hinted handoff queues drained; avoid prolonged single-node outages.","Size max_hints_in_progress to your write throughput.","Use client-side retry policy with exponential backoff for OverloadedException."],"tags":["cassandra","hinted-handoff","overload","write-path"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}