{"record":{"id":"c75d41bc7d934cc1","repo":"apache/cassandra","slug":"growth-of-2f-in-token-ownership-standard-devia","errorCode":null,"errorMessage":"Growth of %.2f%% in token ownership standard deviation after allocating node %d on rack %d above warning threshold of %d%%","messagePattern":"Growth of %\\.2f%% in token ownership standard deviation after allocating node (.+?) on rack (.+?) above warning threshold of (.+?)%%","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/dht/tokenallocator/OfflineTokenAllocator.java","lineNumber":161,"sourceCode":"\n            return new FakeNode(fakeNodeAddressAndPort, rackId, tokens);\n        }\n\n        private void validateAllocation(int nodeId, int rackId)\n        {\n            SummaryStatistics newOwnership = allocation.getAllocationRingOwnership(LOCATION.datacenter, Integer.toString(rackId));\n            SummaryStatistics oldOwnership = lastCheckPoint.put(rackId, newOwnership);\n            if (oldOwnership != null)\n                logger.debug(String.format(\"Replicated node load in rack=%d before allocating node %d: %s.\", rackId, nodeId,\n                                           TokenAllocation.statToString(oldOwnership)));\n            logger.debug(String.format(\"Replicated node load in rack=%d after allocating node %d: %s.\", rackId, nodeId,\n                                       TokenAllocation.statToString(newOwnership)));\n            if (oldOwnership != null && oldOwnership.getStandardDeviation() != 0.0)\n            {\n                double stdDevGrowth = newOwnership.getStandardDeviation() - oldOwnership.getStandardDeviation();\n                if (stdDevGrowth > TokenAllocation.WARN_STDEV_GROWTH)\n                {\n                    logger.warn(String.format(\"Growth of %.2f%% in token ownership standard deviation after allocating node %d on rack %d above warning threshold of %d%%\",\n                                              stdDevGrowth * 100, nodeId, rackId, (int)(TokenAllocation.WARN_STDEV_GROWTH * 100)));\n                }\n            }\n        }\n    }\n\n    private static InetAddressAndPort getLoopbackAddressWithPort(int port)\n    {\n        try\n        {\n            return InetAddressAndPort.getByAddressOverrideDefaults(InetAddress.getByName(\"127.0.0.1\"), port);\n        }\n        catch (UnknownHostException e)\n        {\n            throw new IllegalStateException(\"Unexpected UnknownHostException\", e);\n        }\n    }\n}","sourceCodeStart":143,"sourceCodeEnd":179,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/dht/tokenallocator/OfflineTokenAllocator.java#L143-L179","documentation":"This is a logger.warn, not an exception: during offline token allocation, validateAllocation compares the standard deviation of token ownership before and after allocating tokens for the new node and warns if the standard deviation grew by more than TokenAllocation.WARN_STDEV_GROWTH (10%). It signals that the chosen tokens made load distribution in the rack meaningfully less balanced.","triggerScenarios":"Calling OfflineTokenAllocator.allocateTokensForNode (allocation of N tokens for a node on a given rack) where the resulting ownership distribution's standard deviation exceeds the pre-allocation value by more than the warning threshold.","commonSituations":"Generating tokens offline for a cluster with skewed existing ownership (e.g., previously hand-picked tokens or one big node); allocating into a rack whose load differs greatly from others; allocating many tokens for a small node into an already imbalanced rack.","solutions":["Review the warning in context — if the cluster is already imbalanced, consider re-allocating tokens for the whole cluster offline.","Re-run allocation with the correct rackId and replication strategy so rack-aware balance is computed properly.","Consider letting Cassandra allocate tokens automatically (allocate_tokens_for_local_replication_factor) instead of offline allocation.","Investigate why pre-allocation ownership is skewed (heterogeneous hardware, wrong RF per DC) and rebalance before adding nodes."],"exampleFix":"// before: allocate ignoring rack\nToken[] tokens = OfflineTokenAllocator.instance().allocateTokensForNode(nodeId, metadata, rf, 0);\n// after: allocate with the node's real rack so balance is validated per-rack\nToken[] tokens = OfflineTokenAllocator.instance().allocateTokensForNode(nodeId, metadata, rf, rackId); // then inspect log for stdev growth","handlingStrategy":"validation","validationCode":"OwnershipStats old = TokenAllocation.replicatedOwnership(metadata, snitch, ks, rf);\nif (old.getStandardDeviation() > 0.10) /* rebalance cluster before offline allocation */;","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Check existing ownership balance before offline allocation.","Always pass the correct rackId to allocateTokensForNode.","Prefer built-in RF-aware token allocation over hand-rolled offline allocation.","Treat the warning as input to a full-cluster token reallocation plan."],"tags":["token-allocation","load-balancing","offline-allocator"],"backgroundTag":"load-imbalance-warning","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}