{"record":{"id":"6edf47c10debedf9","repo":"apache/cassandra","slug":"some-hints-were-not-written-before-shutdown-this","errorCode":null,"errorMessage":"Some hints were not written before shutdown.  This is not supposed to happen.  You should (a) run repair, and (b) file a bug report","messagePattern":"Some hints were not written before shutdown\\.  This is not supposed to happen\\.  You should \\(a\\) run repair, and \\(b\\) file a bug report","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/service/StorageProxy.java","lineNumber":3309,"sourceCode":"    public int getMaxHintsInProgress()\n    {\n        return maxHintsInProgress;\n    }\n\n    public void setMaxHintsInProgress(int qs)\n    {\n        maxHintsInProgress = qs;\n    }\n\n    public int getHintsInProgress()\n    {\n        return (int) StorageMetrics.totalHintsInProgress.getCount();\n    }\n\n    public void verifyNoHintsInProgress()\n    {\n        if (getHintsInProgress() > 0)\n            logger.warn(\"Some hints were not written before shutdown.  This is not supposed to happen.  You should (a) run repair, and (b) file a bug report\");\n    }\n\n    private static AtomicInteger getHintsInProgressFor(InetAddressAndPort destination)\n    {\n        try\n        {\n            return hintsInProgress.load(destination);\n        }\n        catch (Exception e)\n        {\n            throw new AssertionError(e);\n        }\n    }\n\n    public static void submitHintForRetryOnDifferentSystem(Mutation mutation)\n    {\n        submitHint(mutation, ImmutableSet.of(HintsService.RETRY_ON_DIFFERENT_SYSTEM_ADDRESS), null);\n    }","sourceCodeStart":3291,"sourceCodeEnd":3327,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/StorageProxy.java#L3291-L3327","documentation":"On shutdown, StorageService calls verifyNoHintsInProgress, which checks totalHintsInProgress. Hints being handed out should have completed before shutdown finishes; a nonzero count means some hint writes did not flush, and the target replicas may miss writes until repaired. Because this 'is not supposed to happen', it is logged as a warning and the message asks the operator to run repair and file a bug.","triggerScenarios":"Node shutdown (decommission, stop, restart) while getHintsInProgress() > 0 — i.e., hint handoff writes were still in flight at verifyNoHintsInProgress time, typically due to a slow/stalled destination or a race in hint dispatch draining.","commonSituations":"Restarting a busy node with many pending hints to an unreachable/slow replica; shutdown timeouts racing hint dispatch threads; known race conditions in hint delivery during rolling restarts.","solutions":["Run repair on the affected ranges after restart (the message's first instruction) to close any missed writes","File a bug report with logs as the message requests, including the destination replicas involved","Avoid restarting while hints backlog is large; check nodetool hints / totalHintsInProgress before planned restarts","Upgrade to the latest patch release — several hint-draining races have been fixed over time"],"exampleFix":"// before\n$ nodetool stop   # hints still in flight -> warning\n\n// after (safe restart procedure)\n$ nodetool tpstats | grep -i hint        # verify no hint activity\n$ nodetool drain && nodetool stop\n$ nodetool repair -pr myks               # after restart","handlingStrategy":"retry","validationCode":"// before planned shutdown\nif (storageProxy.getHintsInProgress() > 0) {\n    waitUntil(() -> storageProxy.getHintsInProgress() == 0, timeout);\n    // only then drain/stop\n}","typeGuard":null,"tryCatchPattern":"try {\n    node.shutdown();\n} finally {\n    if (wasLogged(\"Some hints were not written before shutdown\"))\n        runRepair(node); // close missed hint windows\n}","preventionTips":["Check totalHintsInProgress before restarts; wait for backlog to drain","Avoid restarting while target replicas are down/slow","Keep Cassandra patched — hint-draining races are periodically fixed","Run repair after every unclean shutdown as routine hygiene"],"tags":["hints","shutdown","repair","hints-handoff"],"backgroundTag":"internal-invariant-violation","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}