apache/cassandra · error · ConfigurationException

Following datacenters have active nodes and must be present

Error message

Following datacenters have active nodes and must be present in replication options for keyspace system_auth: %s

What it means

NetworkTopologyStrategy requires that every datacenter containing live nodes be explicitly assigned a replication factor in the keyspace's replication options. When altering or creating the system_auth keyspace (which NetworkTopologyStrategy treats specially), any datacenter with active nodes that is missing from the provided replication options causes this ConfigurationException. This prevents accidentally leaving system_auth unre replicated in a datacenter, which would break authentication for nodes there.

Source

Thrown at src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java:367

    @Override
    public void validateExpectedOptions(ClusterMetadata metadata) throws ConfigurationException
    {
        // Do not accept query with no data centers specified.
        if (this.configOptions.isEmpty())
        {
            throw new ConfigurationException("Configuration for at least one datacenter must be present");
        }

        // Validate the data center names
        super.validateExpectedOptions(metadata);

        if (keyspaceName.equalsIgnoreCase(SchemaConstants.AUTH_KEYSPACE_NAME))
        {
            Set<String> differenceSet = Sets.difference(metadata.directory.knownDatacenters(), configOptions.keySet());
            if (!differenceSet.isEmpty())
            {
                throw new ConfigurationException("Following datacenters have active nodes and must be present in replication options for keyspace " + SchemaConstants.AUTH_KEYSPACE_NAME + ": " + differenceSet.toString());
            }
        }
        logger.info("Configured datacenter replicas are {}", FBUtilities.toString(datacenters));
    }

    @Override
    public void validateOptions() throws ConfigurationException
    {
        for (Entry<String, String> e : this.configOptions.entrySet())
        {
            // prepareOptions should have transformed any "replication_factor" by now
            if (e.getKey().equalsIgnoreCase(REPLICATION_FACTOR))
                throw new ConfigurationException(REPLICATION_FACTOR + " should not appear as an option to NetworkTopologyStrategy");
            validateReplicationFactor(e.getValue());
        }
    }

    @Override

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Add every datacenter containing active nodes to the replication options, e.g. ALTER KEYSPACE system_auth WITH replication = {'class':'NetworkTopologyStrategy','dc1':3,'dc2':3};
2. Run a node/DC health check to confirm which DCs are considered active, then include each one
3. Run repair (rebuild) on system_auth after changing its replication to backfill replicas

Example fix

// before
ALTER KEYSPACE system_auth WITH replication = {'class':'NetworkTopologyStrategy','dc1':3};
// after
ALTER KEYSPACE system_auth WITH replication = {'class':'NetworkTopologyStrategy','dc1':3,'dc2':3};

Set<String> known = StorageService.instance.getTokenMetadata().getAllDatacenters();
Map<String,String> opts = keyspaceReplicationOptions;
List<String> missing = known.stream().filter(dc -> !opts.containsKey(dc)).collect(Collectors.toList());
if (!missing.isEmpty()) throw new IllegalStateException("DCs missing from replication options: " + missing);

try { session.execute("ALTER KEYSPACE system_auth WITH replication = {...}"); }
catch (com.datastax.driver.core.exceptions.InvalidQueryException e) { /* missing DC in options: inspect message and retry with all DCs */ }

When it happens

Trigger: Calling ALTER/CREATE KEYSPACE on system_auth with NetworkTopologyStrategy while at least one DC with active nodes is absent from the replication options map (difference between knownDatacenters and configOptions is non-empty), e.g. after adding a new datacenter to the cluster.

Common situations: Adding a new DC to an existing cluster without updating system_auth replication; running multi-DC setup scripts that only update application keyspaces; bootstrapping a node into a new DC.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Missing replication strategy class
• Your replication factor {} for keyspace {} is higher than th
• default_keyspace_rf (%d) cannot be less than minimum_replica
• default_keyspace_rf cannot be less than 1
• default_keyspace_rf to be set (%d) cannot be less than minim

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/432e9041ff1b03f8. Report an issue: GitHub.