{"record":{"id":"432e9041ff1b03f8","repo":"apache/cassandra","slug":"following-datacenters-have-active-nodes-and-must-b","errorCode":null,"errorMessage":"Following datacenters have active nodes and must be present in replication options for keyspace system_auth: %s","messagePattern":"Following datacenters have active nodes and must be present in replication options for keyspace system_auth: (.+?)","errorType":"validation","errorClass":"ConfigurationException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java","lineNumber":367,"sourceCode":"\n    @Override\n    public void validateExpectedOptions(ClusterMetadata metadata) throws ConfigurationException\n    {\n        // Do not accept query with no data centers specified.\n        if (this.configOptions.isEmpty())\n        {\n            throw new ConfigurationException(\"Configuration for at least one datacenter must be present\");\n        }\n\n        // Validate the data center names\n        super.validateExpectedOptions(metadata);\n\n        if (keyspaceName.equalsIgnoreCase(SchemaConstants.AUTH_KEYSPACE_NAME))\n        {\n            Set<String> differenceSet = Sets.difference(metadata.directory.knownDatacenters(), configOptions.keySet());\n            if (!differenceSet.isEmpty())\n            {\n                throw new ConfigurationException(\"Following datacenters have active nodes and must be present in replication options for keyspace \" + SchemaConstants.AUTH_KEYSPACE_NAME + \": \" + differenceSet.toString());\n            }\n        }\n        logger.info(\"Configured datacenter replicas are {}\", FBUtilities.toString(datacenters));\n    }\n\n    @Override\n    public void validateOptions() throws ConfigurationException\n    {\n        for (Entry<String, String> e : this.configOptions.entrySet())\n        {\n            // prepareOptions should have transformed any \"replication_factor\" by now\n            if (e.getKey().equalsIgnoreCase(REPLICATION_FACTOR))\n                throw new ConfigurationException(REPLICATION_FACTOR + \" should not appear as an option to NetworkTopologyStrategy\");\n            validateReplicationFactor(e.getValue());\n        }\n    }\n\n    @Override","sourceCodeStart":349,"sourceCodeEnd":385,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java#L349-L385","documentation":"NetworkTopologyStrategy requires that every datacenter containing live nodes be explicitly assigned a replication factor in the keyspace's replication options. When altering or creating the system_auth keyspace (which NetworkTopologyStrategy treats specially), any datacenter with active nodes that is missing from the provided replication options causes this ConfigurationException. This prevents accidentally leaving system_auth unre replicated in a datacenter, which would break authentication for nodes there.","triggerScenarios":"Calling ALTER/CREATE KEYSPACE on system_auth with NetworkTopologyStrategy while at least one DC with active nodes is absent from the replication options map (difference between knownDatacenters and configOptions is non-empty), e.g. after adding a new datacenter to the cluster.","commonSituations":"Adding a new DC to an existing cluster without updating system_auth replication; running multi-DC setup scripts that only update application keyspaces; bootstrapping a node into a new DC.","solutions":["Add every datacenter containing active nodes to the replication options, e.g. ALTER KEYSPACE system_auth WITH replication = {'class':'NetworkTopologyStrategy','dc1':3,'dc2':3};","Run a node/DC health check to confirm which DCs are considered active, then include each one","Run repair (rebuild) on system_auth after changing its replication to backfill replicas"],"exampleFix":"// before\nALTER KEYSPACE system_auth WITH replication = {'class':'NetworkTopologyStrategy','dc1':3};\n// after\nALTER KEYSPACE system_auth WITH replication = {'class':'NetworkTopologyStrategy','dc1':3,'dc2':3};","handlingStrategy":"validation","validationCode":"Set<String> known = StorageService.instance.getTokenMetadata().getAllDatacenters();\nMap<String,String> opts = keyspaceReplicationOptions;\nList<String> missing = known.stream().filter(dc -> !opts.containsKey(dc)).collect(Collectors.toList());\nif (!missing.isEmpty()) throw new IllegalStateException(\"DCs missing from replication options: \" + missing);","typeGuard":null,"tryCatchPattern":"try { session.execute(\"ALTER KEYSPACE system_auth WITH replication = {...}\"); }\ncatch (com.datastax.driver.core.exceptions.InvalidQueryException e) { /* missing DC in options: inspect message and retry with all DCs */ }","preventionTips":["After adding a DC, update replication for ALL system keyspaces (system_auth, system_distributed) plus app keyspaces","Enumerate DCs with nodetool status before writing replication maps","Use a cluster-management tool that derives options from knownDatacenters"],"tags":["configuration","replication","keyspace"],"backgroundTag":"missing-required-config-field","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}