apache/pulsar · warning · RestException

Cannot remove cluster ${cluster} from tenant ${tenant}: name

Error message

Cannot remove cluster ${cluster} from tenant ${tenant}: namespace ${ns} still has it as a replication cluster

What it means

HTTP 412 PRECONDITION_FAILED thrown by the tenant-update path: when removing a cluster from a tenant's allowedClusters, the broker iterates every namespace in the tenant and rejects the update if any namespace still lists that cluster in its replication_clusters. This prevents breaking geo-replication by silently removing a replication target still in use.

Source

Thrown at pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java:1089

        Set<String> removedClusters = new java.util.HashSet<>(oldClusters);
        removedClusters.removeAll(newClusters);
        if (removedClusters.isEmpty()) {
            return CompletableFuture.completedFuture(null);
        }

        // For each removed cluster, check if any namespace under this tenant references it
        return tenantResources().getListOfNamespacesAsync(tenant)
                .thenCompose(namespaces -> {
                    java.util.List<CompletableFuture<Void>> checks = new java.util.ArrayList<>();
                    for (String ns : namespaces) {
                        NamespaceName namespaceName = NamespaceName.get(ns);
                        CompletableFuture<Void> check = namespaceResources()
                                .getPoliciesAsync(namespaceName)
                                .thenAccept(policiesOpt -> {
                                    if (policiesOpt.isPresent()) {
                                        for (String cluster : removedClusters) {
                                            if (policiesOpt.get().replication_clusters.contains(cluster)) {
                                                throw new RestException(Status.PRECONDITION_FAILED,
                                                        "Cannot remove cluster " + cluster
                                                                + " from tenant " + tenant
                                                                + ": namespace " + ns
                                                                + " still has it as a replication cluster");
                                            }
                                        }
                                    }
                                });
                        checks.add(check);
                    }
                    return FutureUtil.waitForAll(checks);
                });
    }

    protected PulsarResources getPulsarResources() {
        return pulsar().getPulsarResources();
    }

View on GitHub (pinned to 820761864e)

Solutions

1. Remove the cluster from each namespace's replication_clusters first: admin.namespaces().setNamespaceReplicationClusters(ns, clustersWithoutRemoved)
2. Or delete/unset the cluster from the namespace if replication there is no longer needed
3. Re-run the tenant update after all namespaces are updated
4. List namespaces in the tenant and grep their policies to find which ones reference the removed cluster

Example fix

// before: 412 because ns still replicates to 'east'
admin.tenants().updateTenant("my-tenant", new TenantInfoImpl(adminRoles, Set.of("west")));
// after: remove the cluster from the namespace first
admin.namespaces().setNamespaceReplicationClusters("my-tenant/ns1", Set.of("west"));
admin.tenants().updateTenant("my-tenant", new TenantInfoImpl(adminRoles, Set.of("west")));

Set<String> removing = Set.of("east");
for (String ns : admin.namespaces().getNamespaces(tenant)) {
    if (admin.namespaces().getNamespaceReplicationClusters(ns).containsAll(removing)) {
        throw new IllegalStateException(ns + " still replicates to removed cluster");
    }
}

try {
    admin.tenants().updateTenant(tenant, newInfo);
} catch (PulsarAdminException e) {
    if (e.getStatusCode() == 412 && e.getMessage().contains("still has it as a replication cluster")) {
        throw new IllegalStateException("update namespace replication_clusters before removing the cluster", e);
    } else throw e;
}

When it happens

Trigger: Calling admin.tenants().updateTenant() (PUT /admin/v2/tenants/<tenant>) whose TenantInfo omits a cluster that any of the tenant's namespaces still has in replication_clusters.

Common situations: Decommissioning a data center/cluster: operator drops it from the tenant's allowedClusters while namespaces still replicate there; multi-cluster tenant cleanup scripts that only edit tenant config.

Related errors

• Tenant name or namespace is not valid
• Topic name is not valid
• Partitioned Topic Name should not contain '-partition-'
• Cluster ${cluster} does not exist.
• Exceed maximum number of topics in namespace.

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/971a7dff92608da1. Report an issue: GitHub.