{"record":{"id":"95d1e48e9dcb5535","repo":"apache/cassandra","slug":"not-marking-nodes-down-due-to-local-pause-of-ns","errorCode":null,"errorMessage":"Not marking nodes down due to local pause of {}ns > {}ns","messagePattern":"Not marking nodes down due to local pause of (.+?)ns > (.+?)ns","errorType":"console","errorClass":null,"httpStatus":null,"severity":"info","filePath":"src/java/org/apache/cassandra/gms/FailureDetector.java","lineNumber":376,"sourceCode":"        }\n\n        if (logger.isTraceEnabled() && heartbeatWindow != null)\n            logger.trace(\"Average for {} is {}ns\", ep, heartbeatWindow.mean());\n    }\n\n    public void interpret(InetAddressAndPort ep)\n    {\n        ArrivalWindow hbWnd = arrivalSamples.get(ep);\n        if (hbWnd == null)\n        {\n            return;\n        }\n        long now = preciseTime.now();\n        long diff = now - lastInterpret;\n        lastInterpret = now;\n        if (diff > MAX_LOCAL_PAUSE_IN_NANOS)\n        {\n            logger.warn(\"Not marking nodes down due to local pause of {}ns > {}ns\", diff, MAX_LOCAL_PAUSE_IN_NANOS);\n            lastPause = now;\n            return;\n        }\n        if (preciseTime.now() - lastPause < MAX_LOCAL_PAUSE_IN_NANOS)\n        {\n            logger.debug(\"Still not marking nodes down due to local pause\");\n            return;\n        }\n\n        if (!isAlive(ep))\n            return; // don't convict nodes that are already down - this helps Accord on startup which doesn't report itself alive in Gossip until ready to serve traffic\n\n        double phi = hbWnd.phi(now);\n        logger.trace(\"PHI for {} : {}\", ep, phi);\n\n        if (PHI_FACTOR * phi > getPhiConvictThreshold())\n        {\n            if (logger.isTraceEnabled())","sourceCodeStart":358,"sourceCodeEnd":394,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/gms/FailureDetector.java#L358-L394","documentation":"A logger.warn (not an exception): FailureDetector.interpret skips an entire gossip interpretation round and refuses to mark any nodes DOWN when it detects that the JVM was paused (GC stop-the-world, VM suspend, etc.) for longer than MAX_LOCAL_PAUSE_IN_NANOS since the last run. This prevents mass false-positive node failures after a local pause; failure detection resumes once the pause window has passed.","triggerScenarios":"The interpret() scheduled task runs and observes now - lastInterpret > MAX_LOCAL_PAUSE_IN_NANOS (~2s default, cassandra.max_local_pause_in_ms), typically after a long GC pause or host suspend. The warning fires and nodes are not convicted during that round.","commonSituations":"Long stop-the-world GC pauses (especially with oversized heaps); VM/host live migration or snapshot pauses; cgroup CPU throttling or system suspend on single-node dev environments; laptops going to sleep while running a local cluster.","solutions":["Tune GC to reduce pause lengths (switch to G1/ZGC, right-size heap: recommended 8-16GB, avoid heap sizes causing >2s pauses).","Check gc.log / JVM pause instrumentation to find the source of the pause.","If intentional long pauses are expected, adjust cassandra.max_local_pause_in_ms accordingly (accepting slower failure detection).","Investigate host-level causes (cgroup limits, CPU starvation, virtualization) if no GC pause correlates."],"exampleFix":"// before: JVM_OPTS=\"$JVM_OPTS -Xmx64G\"  // long STW pauses\n// after: cassandra-env.sh — right-size heap / use low-pause collector\nMAX_HEAP_SIZE=\"16G\"\nJVM_OPTS=\"$JVM_OPTS -XX:+UseG1GC\"  // or -XX:+UseZGC on JDK 17+","handlingStrategy":"validation","validationCode":"# before rollout: measure max JVM pause; fail if > max_local_pause_in_ms\nMAX_PAUSE=$(grep 'Total time for which application threads were stopped' gc.log | awk '{print $8}' | sort -n | tail -1)\n[ \"$(echo \"$MAX_PAUSE < 2000\" | bc -l)\" = 1 ] || echo 'GC pauses exceed failure-detection pause window'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Right-size the heap (8-16GB) and use a low-pause collector (G1/ZGC).","Monitor gc.log for pauses approaching the ~2s MAX_LOCAL_PAUSE window.","Avoid host suspend/migration on Cassandra VMs.","Raise cassandra.max_local_pause_in_ms only deliberately, knowing detection slows."],"tags":["failure-detection","gc-pause","gossip"],"backgroundTag":"gc-pause-detected","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}