apache/cassandra · info

Not marking nodes down due to local pause of

Error message

Not marking nodes down due to local pause of {}ns > {}ns

What it means

A logger.warn (not an exception): FailureDetector.interpret skips an entire gossip interpretation round and refuses to mark any nodes DOWN when it detects that the JVM was paused (GC stop-the-world, VM suspend, etc.) for longer than MAX_LOCAL_PAUSE_IN_NANOS since the last run. This prevents mass false-positive node failures after a local pause; failure detection resumes once the pause window has passed.

Solutions

1. Tune GC to reduce pause lengths (switch to G1/ZGC, right-size heap: recommended 8-16GB, avoid heap sizes causing >2s pauses).
2. Check gc.log / JVM pause instrumentation to find the source of the pause.
3. If intentional long pauses are expected, adjust cassandra.max_local_pause_in_ms accordingly (accepting slower failure detection).
4. Investigate host-level causes (cgroup limits, CPU starvation, virtualization) if no GC pause correlates.

Example fix

// before: JVM_OPTS="$JVM_OPTS -Xmx64G"  // long STW pauses
// after: cassandra-env.sh — right-size heap / use low-pause collector
MAX_HEAP_SIZE="16G"
JVM_OPTS="$JVM_OPTS -XX:+UseG1GC"  // or -XX:+UseZGC on JDK 17+

# before rollout: measure max JVM pause; fail if > max_local_pause_in_ms
MAX_PAUSE=$(grep 'Total time for which application threads were stopped' gc.log | awk '{print $8}' | sort -n | tail -1)
[ "$(echo "$MAX_PAUSE < 2000" | bc -l)" = 1 ] || echo 'GC pauses exceed failure-detection pause window'

When it happens

Trigger: The interpret() scheduled task runs and observes now - lastInterpret > MAX_LOCAL_PAUSE_IN_NANOS (~2s default, cassandra.max_local_pause_in_ms), typically after a long GC pause or host suspend. The warning fires and nodes are not convicted during that round.

Common situations: Long stop-the-world GC pauses (especially with oversized heaps); VM/host live migration or snapshot pauses; cgroup CPU throttling or system suspend on single-node dev environments; laptops going to sleep while running a local cluster.

Related errors

• Bad NodeState
• broadcast_address cannot be a wildcard address (
• Can't abort a operation for a node ( ) that is UP - run…
• Can't abort bootstrap for - it is alive
• Can't commit transformations when running in gossip mode…

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/95d1e48e9dcb5535. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/gms/FailureDetector.java:376

        }

        if (logger.isTraceEnabled() && heartbeatWindow != null)
            logger.trace("Average for {} is {}ns", ep, heartbeatWindow.mean());
    }

    public void interpret(InetAddressAndPort ep)
    {
        ArrivalWindow hbWnd = arrivalSamples.get(ep);
        if (hbWnd == null)
        {
            return;
        }
        long now = preciseTime.now();
        long diff = now - lastInterpret;
        lastInterpret = now;
        if (diff > MAX_LOCAL_PAUSE_IN_NANOS)
        {
            logger.warn("Not marking nodes down due to local pause of {}ns > {}ns", diff, MAX_LOCAL_PAUSE_IN_NANOS);
            lastPause = now;
            return;
        }
        if (preciseTime.now() - lastPause < MAX_LOCAL_PAUSE_IN_NANOS)
        {
            logger.debug("Still not marking nodes down due to local pause");
            return;
        }

        if (!isAlive(ep))
            return; // don't convict nodes that are already down - this helps Accord on startup which doesn't report itself alive in Gossip until ready to serve traffic

        double phi = hbWnd.phi(now);
        logger.trace("PHI for {} : {}", ep, phi);

        if (PHI_FACTOR * phi > getPhiConvictThreshold())
        {
            if (logger.isTraceEnabled())

View on GitHub (pinned to 88fd0f6a0e)