{"record":{"id":"971a7dff92608da1","repo":"apache/pulsar","slug":"cannot-remove-cluster-cluster-from-tenant-ten","errorCode":null,"errorMessage":"Cannot remove cluster ${cluster} from tenant ${tenant}: namespace ${ns} still has it as a replication cluster","messagePattern":"Cannot remove cluster (.+?) from tenant (.+?): namespace (.+?) still has it as a replication cluster","errorType":"http","errorClass":"RestException","httpStatus":412,"severity":"warning","filePath":"pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java","lineNumber":1089,"sourceCode":"        Set<String> removedClusters = new java.util.HashSet<>(oldClusters);\n        removedClusters.removeAll(newClusters);\n        if (removedClusters.isEmpty()) {\n            return CompletableFuture.completedFuture(null);\n        }\n\n        // For each removed cluster, check if any namespace under this tenant references it\n        return tenantResources().getListOfNamespacesAsync(tenant)\n                .thenCompose(namespaces -> {\n                    java.util.List<CompletableFuture<Void>> checks = new java.util.ArrayList<>();\n                    for (String ns : namespaces) {\n                        NamespaceName namespaceName = NamespaceName.get(ns);\n                        CompletableFuture<Void> check = namespaceResources()\n                                .getPoliciesAsync(namespaceName)\n                                .thenAccept(policiesOpt -> {\n                                    if (policiesOpt.isPresent()) {\n                                        for (String cluster : removedClusters) {\n                                            if (policiesOpt.get().replication_clusters.contains(cluster)) {\n                                                throw new RestException(Status.PRECONDITION_FAILED,\n                                                        \"Cannot remove cluster \" + cluster\n                                                                + \" from tenant \" + tenant\n                                                                + \": namespace \" + ns\n                                                                + \" still has it as a replication cluster\");\n                                            }\n                                        }\n                                    }\n                                });\n                        checks.add(check);\n                    }\n                    return FutureUtil.waitForAll(checks);\n                });\n    }\n\n    protected PulsarResources getPulsarResources() {\n        return pulsar().getPulsarResources();\n    }\n","sourceCodeStart":1071,"sourceCodeEnd":1107,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java#L1071-L1107","documentation":"HTTP 412 PRECONDITION_FAILED thrown by the tenant-update path: when removing a cluster from a tenant's allowedClusters, the broker iterates every namespace in the tenant and rejects the update if any namespace still lists that cluster in its replication_clusters. This prevents breaking geo-replication by silently removing a replication target still in use.","triggerScenarios":"Calling admin.tenants().updateTenant() (PUT /admin/v2/tenants/<tenant>) whose TenantInfo omits a cluster that any of the tenant's namespaces still has in replication_clusters.","commonSituations":"Decommissioning a data center/cluster: operator drops it from the tenant's allowedClusters while namespaces still replicate there; multi-cluster tenant cleanup scripts that only edit tenant config.","solutions":["Remove the cluster from each namespace's replication_clusters first: admin.namespaces().setNamespaceReplicationClusters(ns, clustersWithoutRemoved)","Or delete/unset the cluster from the namespace if replication there is no longer needed","Re-run the tenant update after all namespaces are updated","List namespaces in the tenant and grep their policies to find which ones reference the removed cluster"],"exampleFix":"// before: 412 because ns still replicates to 'east'\nadmin.tenants().updateTenant(\"my-tenant\", new TenantInfoImpl(adminRoles, Set.of(\"west\")));\n// after: remove the cluster from the namespace first\nadmin.namespaces().setNamespaceReplicationClusters(\"my-tenant/ns1\", Set.of(\"west\"));\nadmin.tenants().updateTenant(\"my-tenant\", new TenantInfoImpl(adminRoles, Set.of(\"west\")));","handlingStrategy":"validation","validationCode":"Set<String> removing = Set.of(\"east\");\nfor (String ns : admin.namespaces().getNamespaces(tenant)) {\n    if (admin.namespaces().getNamespaceReplicationClusters(ns).containsAll(removing)) {\n        throw new IllegalStateException(ns + \" still replicates to removed cluster\");\n    }\n}","typeGuard":null,"tryCatchPattern":"try {\n    admin.tenants().updateTenant(tenant, newInfo);\n} catch (PulsarAdminException e) {\n    if (e.getStatusCode() == 412 && e.getMessage().contains(\"still has it as a replication cluster\")) {\n        throw new IllegalStateException(\"update namespace replication_clusters before removing the cluster\", e);\n    } else throw e;\n}","preventionTips":["Always update namespaces' replication_clusters before shrinking tenant allowedClusters","Automate namespace scanning when modifying tenant clusters","Treat 412 as a workflow-ordering error, not a transient failure"],"tags":["http-412","tenant","replication-clusters","geo-replication"],"backgroundTag":"dependent-resource-in-use","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}