Gossip stage has pending tasks; skipping status check (no…

Error message

Gossip stage has {} pending tasks; skipping status check (no nodes will be marked down)

What it means

Gossiper's periodic status check (failure detection) skips a round when the GossipStage executor is backlogged: if pending tasks exceed a limit and the last processed message is more than 1 second old, it warns and returns without marking any nodes down. This prevents false failure detection caused by local overload rather than actual peer failure.

Solutions

Investigate GossipStage backlogs: check thread pool metrics and increase gossip hardware/threads if persistently saturated.
Rule out long GC pauses or CPU contention on the node (check gc logs, steal time).
Stagger rolling operations (bootstrap/decommission) to reduce gossip message bursts.
Verify no debugging/trace logging on Gossiper is slowing message processing.

Example fix

// before: overloading one node with concurrent bootstraps
// after: pace operations
nodetool bootstrap nodeA; sleep 300; nodetool bootstrap nodeB;

Defensive patterns

Strategy: retry

Validate before calling

// monitor backlog before operations
long pending = Stage.GOSSIP.getPendingTaskCount();

Prevention

Watch GossipStage pending/completed metrics; alert on sustained backlog.
Avoid heavy trace logging on Gossiper in production.
Scale gossip threads and avoid simultaneous restarts/bootstraps in large clusters.
Keep GC pauses short (large heaps: use G1/ZGC appropriately).

When it happens

Trigger: GossipStage pending task count exceeds the configured threshold while lastProcessedMessageAt is over 1000ms behind during doStatusCheck's periodic run.

Common situations: Large clusters with heavy gossip traffic; CPU starvation or long GC pauses; bursts of gossip messages (e.g., many nodes joining at once); thread pool misconfiguration reducing GossipStage throughput.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/a9329972f78c5a8a. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/gms/Gossiper.java:979

    @VisibleForTesting
    void doStatusCheck()
    {
        logger.trace("Performing status check ...");

        long now = currentTimeMillis();
        long nowNano = nanoTime();

        long pending = Stage.GOSSIP.executor().getPendingTaskCount();
        if (pending > 0 && lastProcessedMessageAt < now - 1000)
        {
            // if some new messages just arrived, give the executor some time to work on them
            Uninterruptibles.sleepUninterruptibly(100, TimeUnit.MILLISECONDS);

            // still behind?  something's broke
            if (lastProcessedMessageAt < now - 1000)
            {
                logger.warn("Gossip stage has {} pending tasks; skipping status check (no nodes will be marked down)", pending);
                return;
            }
        }

        ClusterMetadata metadata = ClusterMetadata.current();
        Set<InetAddressAndPort> eps = endpointStateMap.keySet();
        for (InetAddressAndPort endpoint : eps)
        {
            if (endpoint.equals(getBroadcastAddressAndPort()))
                continue;

            FailureDetector.instance.interpret(endpoint);
            EndpointState epState = endpointStateMap.get(endpoint);
            if (epState != null)
            {
                // check if this is a fat client. fat clients are removed automatically from
                // gossip after FatClientTimeout.  Do not remove dead states here.
                if (isGossipOnlyMember(endpoint)

View on GitHub (pinned to 88fd0f6a0e)