apache/cassandra · critical · UnrecoverableIllegalStateException

Internode messaging byte limits that are shared between…

Error message

Internode messaging byte limits that are shared between connections is invalid (using=+using+)

What it means

Concurrent side of ResourceLimits.Release: an outbound connection released more bytes than it had allocated against the shared endpoint/global limit, which would corrupt the accounting. Because recovery would require tearing down and reinitializing all connections sharing the limit, the JVM is deliberately terminated via UnrecoverableIllegalStateException.

Solutions

Let the JVM terminate and rely on the external supervisor to restart Cassandra into a known-good state
Collect logs/core dump and report the accounting bug (double-release) to the project with the node's messaging logs
Check for unauthorized patches or version mismatches in the messaging/connection code

Defensive patterns

Strategy: retry

Try / catch

try { limits.release(bytes); } catch (UnrecoverableIllegalStateException e) { /* do NOT swallow: JVM is going down; flush logs and alert ops */ }

Prevention

Do not patch connection release/allocate accounting without tests
Pair every allocate with exactly one release in outbound connections
Run the negativeConcurrentUsingValueKillsJVMTest when touching ResourceLimits

When it happens

Trigger: A bug in outbound connection accounting (double release, release after close, missed allocation) makes using go negative when OutboundConnectionLimit.release() runs on a shared (endpoint/global) limit.

Common situations: Hit during stress with many internode connections; typically indicates a Cassandra-internal bug, flaky NIC/connection teardown races, or a patched/modified messaging layer. This path is exercised by negativeConcurrentUsingValueKillsJVMTest.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/e382c7ce2632b55a. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/net/ResourceLimits.java:168

            {
                current = using;
                next = current + amount;
            } while (!usingUpdater.compareAndSet(this, current, next));
        }

        public Outcome release(long amount)
        {
            assert amount >= 0;
            long using = usingUpdater.addAndGet(this, -amount);
            if (using < 0L)
            {
                // Should never be able to release more than was allocated.  While recovery is
                // possible it would require synchronizing the closing of all outbound connections
                // and reinitializing the Concurrent limit before reopening.  For such an unlikely path
                // (previously this was an assert), it is safer to terminate the JVM and have something external
                // restart and get back to a known good state rather than intermittently crashing on any of
                // the connections sharing this limit.
                throw new UnrecoverableIllegalStateException(
                    "Internode messaging byte limits that are shared between connections is invalid (using="+using+")");
            }
            return using >= limit ? Outcome.ABOVE_LIMIT : Outcome.BELOW_LIMIT;
        }
    }

    /**
     * A cheaper, thread-unsafe permit container to be used for unshared limits.
     */
    public static class Basic implements Limit
    {
        private long limit;
        private long using;

        public Basic(long limit)
        {
            this.limit = limit;
        }

View on GitHub (pinned to 88fd0f6a0e)