Growth of %.2f%% in token ownership standard deviation…

Error message

Growth of %.2f%% in token ownership standard deviation after allocating node %d on rack %d above warning threshold of %d%%

What it means

This is a logger.warn, not an exception: during offline token allocation, validateAllocation compares the standard deviation of token ownership before and after allocating tokens for the new node and warns if the standard deviation grew by more than TokenAllocation.WARN_STDEV_GROWTH (10%). It signals that the chosen tokens made load distribution in the rack meaningfully less balanced.

Solutions

Review the warning in context — if the cluster is already imbalanced, consider re-allocating tokens for the whole cluster offline.
Re-run allocation with the correct rackId and replication strategy so rack-aware balance is computed properly.
Consider letting Cassandra allocate tokens automatically (allocate_tokens_for_local_replication_factor) instead of offline allocation.
Investigate why pre-allocation ownership is skewed (heterogeneous hardware, wrong RF per DC) and rebalance before adding nodes.

Example fix

// before: allocate ignoring rack
Token[] tokens = OfflineTokenAllocator.instance().allocateTokensForNode(nodeId, metadata, rf, 0);
// after: allocate with the node's real rack so balance is validated per-rack
Token[] tokens = OfflineTokenAllocator.instance().allocateTokensForNode(nodeId, metadata, rf, rackId); // then inspect log for stdev growth

Defensive patterns

Strategy: validation

Validate before calling

OwnershipStats old = TokenAllocation.replicatedOwnership(metadata, snitch, ks, rf);
if (old.getStandardDeviation() > 0.10) /* rebalance cluster before offline allocation */;

Prevention

Check existing ownership balance before offline allocation.
Always pass the correct rackId to allocateTokensForNode.
Prefer built-in RF-aware token allocation over hand-rolled offline allocation.
Treat the warning as input to a full-cluster token reallocation plan.

When it happens

Trigger: Calling OfflineTokenAllocator.allocateTokensForNode (allocation of N tokens for a node on a given rack) where the resulting ownership distribution's standard deviation exceeds the pre-allocation value by more than the warning threshold.

Common situations: Generating tokens offline for a cluster with skewed existing ownership (e.g., previously hand-picked tokens or one big node); allocating into a rack whose load differs greatly from others; allocating many tokens for a small node into an already imbalanced rack.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/c75d41bc7d934cc1. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/dht/tokenallocator/OfflineTokenAllocator.java:161

            return new FakeNode(fakeNodeAddressAndPort, rackId, tokens);
        }

        private void validateAllocation(int nodeId, int rackId)
        {
            SummaryStatistics newOwnership = allocation.getAllocationRingOwnership(LOCATION.datacenter, Integer.toString(rackId));
            SummaryStatistics oldOwnership = lastCheckPoint.put(rackId, newOwnership);
            if (oldOwnership != null)
                logger.debug(String.format("Replicated node load in rack=%d before allocating node %d: %s.", rackId, nodeId,
                                           TokenAllocation.statToString(oldOwnership)));
            logger.debug(String.format("Replicated node load in rack=%d after allocating node %d: %s.", rackId, nodeId,
                                       TokenAllocation.statToString(newOwnership)));
            if (oldOwnership != null && oldOwnership.getStandardDeviation() != 0.0)
            {
                double stdDevGrowth = newOwnership.getStandardDeviation() - oldOwnership.getStandardDeviation();
                if (stdDevGrowth > TokenAllocation.WARN_STDEV_GROWTH)
                {
                    logger.warn(String.format("Growth of %.2f%% in token ownership standard deviation after allocating node %d on rack %d above warning threshold of %d%%",
                                              stdDevGrowth * 100, nodeId, rackId, (int)(TokenAllocation.WARN_STDEV_GROWTH * 100)));
                }
            }
        }
    }

    private static InetAddressAndPort getLoopbackAddressWithPort(int port)
    {
        try
        {
            return InetAddressAndPort.getByAddressOverrideDefaults(InetAddress.getByName("127.0.0.1"), port);
        }
        catch (UnknownHostException e)
        {
            throw new IllegalStateException("Unexpected UnknownHostException", e);
        }
    }
}

View on GitHub (pinned to 88fd0f6a0e)