{"record":{"id":"e382c7ce2632b55a","repo":"apache/cassandra","slug":"internode-messaging-byte-limits-that-are-shared-be","errorCode":null,"errorMessage":"Internode messaging byte limits that are shared between connections is invalid (using=+using+)","messagePattern":"Internode messaging byte limits that are shared between connections is invalid \\(using=\\+using\\+\\)","errorType":"exception","errorClass":"UnrecoverableIllegalStateException","httpStatus":null,"severity":"critical","filePath":"src/java/org/apache/cassandra/net/ResourceLimits.java","lineNumber":168,"sourceCode":"            {\n                current = using;\n                next = current + amount;\n            } while (!usingUpdater.compareAndSet(this, current, next));\n        }\n\n        public Outcome release(long amount)\n        {\n            assert amount >= 0;\n            long using = usingUpdater.addAndGet(this, -amount);\n            if (using < 0L)\n            {\n                // Should never be able to release more than was allocated.  While recovery is\n                // possible it would require synchronizing the closing of all outbound connections\n                // and reinitializing the Concurrent limit before reopening.  For such an unlikely path\n                // (previously this was an assert), it is safer to terminate the JVM and have something external\n                // restart and get back to a known good state rather than intermittently crashing on any of\n                // the connections sharing this limit.\n                throw new UnrecoverableIllegalStateException(\n                    \"Internode messaging byte limits that are shared between connections is invalid (using=\"+using+\")\");\n            }\n            return using >= limit ? Outcome.ABOVE_LIMIT : Outcome.BELOW_LIMIT;\n        }\n    }\n\n    /**\n     * A cheaper, thread-unsafe permit container to be used for unshared limits.\n     */\n    public static class Basic implements Limit\n    {\n        private long limit;\n        private long using;\n\n        public Basic(long limit)\n        {\n            this.limit = limit;\n        }","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/net/ResourceLimits.java#L150-L186","documentation":"Concurrent side of ResourceLimits.Release: an outbound connection released more bytes than it had allocated against the shared endpoint/global limit, which would corrupt the accounting. Because recovery would require tearing down and reinitializing all connections sharing the limit, the JVM is deliberately terminated via UnrecoverableIllegalStateException.","triggerScenarios":"A bug in outbound connection accounting (double release, release after close, missed allocation) makes using go negative when OutboundConnectionLimit.release() runs on a shared (endpoint/global) limit.","commonSituations":"Hit during stress with many internode connections; typically indicates a Cassandra-internal bug, flaky NIC/connection teardown races, or a patched/modified messaging layer. This path is exercised by negativeConcurrentUsingValueKillsJVMTest.","solutions":["Let the JVM terminate and rely on the external supervisor to restart Cassandra into a known-good state","Collect logs/core dump and report the accounting bug (double-release) to the project with the node's messaging logs","Check for unauthorized patches or version mismatches in the messaging/connection code"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { limits.release(bytes); } catch (UnrecoverableIllegalStateException e) { /* do NOT swallow: JVM is going down; flush logs and alert ops */ }","preventionTips":["Do not patch connection release/allocate accounting without tests","Pair every allocate with exactly one release in outbound connections","Run the negativeConcurrentUsingValueKillsJVMTest when touching ResourceLimits"],"tags":["network","resource-limits","jvm-exit","concurrency"],"backgroundTag":"internal-invariant-violation","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}