{"record":{"id":"ad46b47adeb3b44a","repo":"apache/pulsar","slug":"failed-to-validate-global-cluster-configuration-ad46b4","errorCode":null,"errorMessage":"Failed to validate global cluster configuration : ns=%s  emsg=%s","messagePattern":"Failed to validate global cluster configuration : ns=(.+?)  emsg=(.+?)","errorType":"http","errorClass":"RestException","httpStatus":503,"severity":"error","filePath":"pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java","lineNumber":801,"sourceCode":"        try {\n            ClusterDataImpl peerClusterData = checkLocalOrGetPeerReplicationCluster(pulsar(), namespace)\n                    .get(timeout, SECONDS);\n            // if peer-cluster-data is present it means namespace is owned by that peer-cluster and request should be\n            // redirect to the peer-cluster\n            if (peerClusterData != null) {\n                URI redirect = getRedirectionUrl(peerClusterData);\n                // redirect to the cluster requested\n                    log.debug()\n                            .attr(\"redirect\", redirect)\n                            .attr(\"cluster\", peerClusterData)\n                            .log(\"Redirecting the rest call\");\n\n                                throw new WebApplicationException(Response.temporaryRedirect(redirect).build());\n            }\n        } catch (InterruptedException e) {\n            log.warn().attr(\"timeoutSec\", timeout).attr(\"namespace\", namespace)\n                    .log(\"Timeout while validating policy\");\n            throw new RestException(Status.SERVICE_UNAVAILABLE, String.format(\n                    \"Failed to validate global cluster configuration : ns=%s  emsg=%s\", namespace, e.getMessage()));\n        } catch (WebApplicationException e) {\n            throw e;\n        } catch (Exception e) {\n            Throwable throwable = FutureUtil.unwrapCompletionException(e);\n            if (throwable instanceof WebApplicationException webApplicationException) {\n                throw webApplicationException;\n            }\n            throw new RestException(Status.SERVICE_UNAVAILABLE, String.format(\n                    \"Failed to validate global cluster configuration : ns=%s  emsg=%s\", namespace, e.getMessage()));\n        }\n    }\n\n    protected CompletableFuture<Void> validateGlobalNamespaceOwnershipAsync(NamespaceName namespace) {\n        return checkLocalOrGetPeerReplicationCluster(pulsar(), namespace)\n                .thenAccept(peerClusterData -> {\n                    // if peer-cluster-data is present it means namespace is owned by that peer-cluster and request\n                    // should be redirect to the peer-cluster","sourceCodeStart":783,"sourceCodeEnd":819,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java#L783-L819","documentation":"Thrown in validateGlobalNamespaceOwnership when the async ownership/peer-replication-cluster check is interrupted while waiting, producing HTTP 503 SERVICE_UNAVAILABLE. The broker could not confirm in time whether the local cluster owns the namespace or which peer cluster should serve it, so the admin API request fails rather than serving stale routing.","triggerScenarios":"Any admin REST call routed through validateGlobalNamespaceOwnership (e.g. topic lookup, namespace operations on a global namespace) where checkLocalOrGetPeerReplicationCluster times out waiting on the configuration cache store, or the waiting thread is interrupted.","commonSituations":"Slow or overloaded metadata store (ZooKeeper/etcd) during broker startup; network partition between broker and config cluster; heavy namespace bundle load causing ownership lookups to exceed the validation timeout.","solutions":["Check broker-to-metadata-store connectivity and latency; fix the underlying store slowness or partition","Retry the request once the metadata store is healthy — this is a transient 503","Increase the broker's ownership-validation timeout configuration if timeouts occur under sustained load","Verify the namespace's replication clusters are resolvable and present in the cluster configuration"],"exampleFix":"// before: intermittent 503 failures during lookup\nclient.lookup().getTopic(topicName);\n// after: retry on 503 with backoff\nCompletableFuture.supplyAsync(() -> client.lookup().getTopic(topicName))\n    .orTimeout(30, TimeUnit.SECONDS)\n    .exceptionallyCompose(e -> (e instanceof PulsarClientException && String.valueOf(e).contains(\"503\"))\n        ? CompletableFuture.completedFuture(client.lookup().getTopic(topicName))\n        : CompletableFuture.failedFuture(e));","handlingStrategy":"retry","validationCode":"// precheck: ensure the namespace's replication cluster is registered and reachable\nClusterData cd = admin.clusters().getCluster(namespaceCluster);\nif (cd.getServiceUrl() == null) throw new IllegalStateException(\"peer cluster missing serviceUrl\");","typeGuard":null,"tryCatchPattern":"try {\n    doAdminCall();\n} catch (PulsarAdminException e) {\n    if (e.getStatusCode() == 503 && e.getMessage().contains(\"Failed to validate global cluster configuration\")) {\n        // transient metadata-store timeout: retry with backoff\n        Thread.sleep(retryDelayMs);\n        doAdminCall();\n    } else throw e;\n}","preventionTips":["Monitor metadata-store latency and quorum health","Retry idempotent admin calls on 503 with exponential backoff","Keep namespace replication-cluster lists small and valid"],"tags":["http-503","metadata-store","timeout","namespace-ownership"],"backgroundTag":"metadata-store-timeout","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}