apache/pulsar · error · RestException

Failed to validate global cluster configuration : ns=%s ems

Error message

Failed to validate global cluster configuration : ns=%s  emsg=%s

What it means

Thrown in validateGlobalNamespaceOwnership when the async ownership/peer-replication-cluster check is interrupted while waiting, producing HTTP 503 SERVICE_UNAVAILABLE. The broker could not confirm in time whether the local cluster owns the namespace or which peer cluster should serve it, so the admin API request fails rather than serving stale routing.

Source

Thrown at pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java:801

        try {
            ClusterDataImpl peerClusterData = checkLocalOrGetPeerReplicationCluster(pulsar(), namespace)
                    .get(timeout, SECONDS);
            // if peer-cluster-data is present it means namespace is owned by that peer-cluster and request should be
            // redirect to the peer-cluster
            if (peerClusterData != null) {
                URI redirect = getRedirectionUrl(peerClusterData);
                // redirect to the cluster requested
                    log.debug()
                            .attr("redirect", redirect)
                            .attr("cluster", peerClusterData)
                            .log("Redirecting the rest call");

                                throw new WebApplicationException(Response.temporaryRedirect(redirect).build());
            }
        } catch (InterruptedException e) {
            log.warn().attr("timeoutSec", timeout).attr("namespace", namespace)
                    .log("Timeout while validating policy");
            throw new RestException(Status.SERVICE_UNAVAILABLE, String.format(
                    "Failed to validate global cluster configuration : ns=%s  emsg=%s", namespace, e.getMessage()));
        } catch (WebApplicationException e) {
            throw e;
        } catch (Exception e) {
            Throwable throwable = FutureUtil.unwrapCompletionException(e);
            if (throwable instanceof WebApplicationException webApplicationException) {
                throw webApplicationException;
            }
            throw new RestException(Status.SERVICE_UNAVAILABLE, String.format(
                    "Failed to validate global cluster configuration : ns=%s  emsg=%s", namespace, e.getMessage()));
        }
    }

    protected CompletableFuture<Void> validateGlobalNamespaceOwnershipAsync(NamespaceName namespace) {
        return checkLocalOrGetPeerReplicationCluster(pulsar(), namespace)
                .thenAccept(peerClusterData -> {
                    // if peer-cluster-data is present it means namespace is owned by that peer-cluster and request
                    // should be redirect to the peer-cluster

View on GitHub (pinned to 820761864e)

Solutions

1. Check broker-to-metadata-store connectivity and latency; fix the underlying store slowness or partition
2. Retry the request once the metadata store is healthy — this is a transient 503
3. Increase the broker's ownership-validation timeout configuration if timeouts occur under sustained load
4. Verify the namespace's replication clusters are resolvable and present in the cluster configuration

Example fix

// before: intermittent 503 failures during lookup
client.lookup().getTopic(topicName);
// after: retry on 503 with backoff
CompletableFuture.supplyAsync(() -> client.lookup().getTopic(topicName))
    .orTimeout(30, TimeUnit.SECONDS)
    .exceptionallyCompose(e -> (e instanceof PulsarClientException && String.valueOf(e).contains("503"))
        ? CompletableFuture.completedFuture(client.lookup().getTopic(topicName))
        : CompletableFuture.failedFuture(e));

// precheck: ensure the namespace's replication cluster is registered and reachable
ClusterData cd = admin.clusters().getCluster(namespaceCluster);
if (cd.getServiceUrl() == null) throw new IllegalStateException("peer cluster missing serviceUrl");

try {
    doAdminCall();
} catch (PulsarAdminException e) {
    if (e.getStatusCode() == 503 && e.getMessage().contains("Failed to validate global cluster configuration")) {
        // transient metadata-store timeout: retry with backoff
        Thread.sleep(retryDelayMs);
        doAdminCall();
    } else throw e;
}

When it happens

Trigger: Any admin REST call routed through validateGlobalNamespaceOwnership (e.g. topic lookup, namespace operations on a global namespace) where checkLocalOrGetPeerReplicationCluster times out waiting on the configuration cache store, or the waiting thread is interrupted.

Common situations: Slow or overloaded metadata store (ZooKeeper/etcd) during broker startup; network partition between broker and config cluster; heavy namespace bundle load causing ownership lookups to exceed the validation timeout.

Related errors

• Failed to validate global cluster configuration
• RestException(e)
• Failed to refresh self owner info.
• Time-out while checking authorization
• Failed to get children of ${path}

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/ad46b47adeb3b44a. Report an issue: GitHub.