{"record":{"id":"a9329972f78c5a8a","repo":"apache/cassandra","slug":"gossip-stage-has-pending-tasks-skipping-status","errorCode":null,"errorMessage":"Gossip stage has {} pending tasks; skipping status check (no nodes will be marked down)","messagePattern":"Gossip stage has (.+?) pending tasks; skipping status check \\(no nodes will be marked down\\)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/gms/Gossiper.java","lineNumber":979,"sourceCode":"\n    @VisibleForTesting\n    void doStatusCheck()\n    {\n        logger.trace(\"Performing status check ...\");\n\n        long now = currentTimeMillis();\n        long nowNano = nanoTime();\n\n        long pending = Stage.GOSSIP.executor().getPendingTaskCount();\n        if (pending > 0 && lastProcessedMessageAt < now - 1000)\n        {\n            // if some new messages just arrived, give the executor some time to work on them\n            Uninterruptibles.sleepUninterruptibly(100, TimeUnit.MILLISECONDS);\n\n            // still behind?  something's broke\n            if (lastProcessedMessageAt < now - 1000)\n            {\n                logger.warn(\"Gossip stage has {} pending tasks; skipping status check (no nodes will be marked down)\", pending);\n                return;\n            }\n        }\n\n        ClusterMetadata metadata = ClusterMetadata.current();\n        Set<InetAddressAndPort> eps = endpointStateMap.keySet();\n        for (InetAddressAndPort endpoint : eps)\n        {\n            if (endpoint.equals(getBroadcastAddressAndPort()))\n                continue;\n\n            FailureDetector.instance.interpret(endpoint);\n            EndpointState epState = endpointStateMap.get(endpoint);\n            if (epState != null)\n            {\n                // check if this is a fat client. fat clients are removed automatically from\n                // gossip after FatClientTimeout.  Do not remove dead states here.\n                if (isGossipOnlyMember(endpoint)","sourceCodeStart":961,"sourceCodeEnd":997,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/gms/Gossiper.java#L961-L997","documentation":"Gossiper's periodic status check (failure detection) skips a round when the GossipStage executor is backlogged: if pending tasks exceed a limit and the last processed message is more than 1 second old, it warns and returns without marking any nodes down. This prevents false failure detection caused by local overload rather than actual peer failure.","triggerScenarios":"GossipStage pending task count exceeds the configured threshold while lastProcessedMessageAt is over 1000ms behind during doStatusCheck's periodic run.","commonSituations":"Large clusters with heavy gossip traffic; CPU starvation or long GC pauses; bursts of gossip messages (e.g., many nodes joining at once); thread pool misconfiguration reducing GossipStage throughput.","solutions":["Investigate GossipStage backlogs: check thread pool metrics and increase gossip hardware/threads if persistently saturated.","Rule out long GC pauses or CPU contention on the node (check gc logs, steal time).","Stagger rolling operations (bootstrap/decommission) to reduce gossip message bursts.","Verify no debugging/trace logging on Gossiper is slowing message processing."],"exampleFix":"// before: overloading one node with concurrent bootstraps\n// after: pace operations\nnodetool bootstrap nodeA; sleep 300; nodetool bootstrap nodeB;","handlingStrategy":"retry","validationCode":"// monitor backlog before operations\nlong pending = Stage.GOSSIP.getPendingTaskCount();","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Watch GossipStage pending/completed metrics; alert on sustained backlog.","Avoid heavy trace logging on Gossiper in production.","Scale gossip threads and avoid simultaneous restarts/bootstraps in large clusters.","Keep GC pauses short (large heaps: use G1/ZGC appropriately)."],"tags":["cassandra","gossip","backpressure","thread-pool"],"backgroundTag":"gossip-stage-backlog","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}