{"record":{"id":"9d770c8251911585","repo":"temporalio/temporal","slug":"bug-found-in-cluster-metadata-with-error-v","errorCode":null,"errorMessage":"Bug found in cluster metadata with error %v","messagePattern":"Bug found in cluster metadata with error (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"service/worker/replicator/replicator.go","lineNumber":160,"sourceCode":"\t\t) {\n\t\t\tcurrentClusterName := r.clusterMetadata.GetCurrentClusterName()\n\t\t\tr.namespaceProcessorsLock.Lock()\n\t\t\tdefer r.namespaceProcessorsLock.Unlock()\n\t\t\tfor clusterName := range newClusterMetadata {\n\t\t\t\tif clusterName == currentClusterName {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tif processor, ok := r.namespaceProcessors[clusterName]; ok {\n\t\t\t\t\tprocessor.Stop()\n\t\t\t\t\tdelete(r.namespaceProcessors, clusterName)\n\t\t\t\t}\n\n\t\t\t\tif clusterInfo := newClusterMetadata[clusterName]; clusterInfo != nil && clusterInfo.Enabled {\n\t\t\t\t\tremoteAdminClient, err := r.clientBean.GetRemoteAdminClient(clusterName)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\t// Cannot find remote cluster info.\n\t\t\t\t\t\t// This should never happen as cluster metadata should have the up-to-date data.\n\t\t\t\t\t\tpanic(fmt.Sprintf(\"Bug found in cluster metadata with error %v\", err))\n\t\t\t\t\t}\n\t\t\t\t\tprocessor := newReplicationMessageProcessor(\n\t\t\t\t\t\tcurrentClusterName,\n\t\t\t\t\t\tclusterName,\n\t\t\t\t\t\tlog.With(r.logger, tag.ComponentReplicationTaskProcessor, tag.SourceCluster(clusterName)),\n\t\t\t\t\t\tr.eventLogger,\n\t\t\t\t\t\tr.emitNamespaceLifecycleEvents,\n\t\t\t\t\t\tr.eventDataProvider,\n\t\t\t\t\t\tremoteAdminClient,\n\t\t\t\t\t\tr.metricsHandler,\n\t\t\t\t\t\tr.namespaceReplicationTaskExecutor,\n\t\t\t\t\t\tr.customTaskHandler,\n\t\t\t\t\t\tr.hostInfo,\n\t\t\t\t\t\tr.serviceResolver,\n\t\t\t\t\t\tr.namespaceReplicationQueue,\n\t\t\t\t\t\tr.matchingClient,\n\t\t\t\t\t\tr.namespaceRegistry,\n\t\t\t\t\t)","sourceCodeStart":142,"sourceCodeEnd":178,"githubUrl":"https://github.com/temporalio/temporal/blob/bde624efd13fbd3843654058db6d9c716166318b/service/worker/replicator/replicator.go#L142-L178","documentation":"This panic fires in the Temporal history replication replicator when a metadata-change callback sees a remote cluster marked Enabled but fails to obtain a remote admin gRPC client for it. The invariant is that cluster metadata (persisted + membership) should always contain a resolvable admin address for any enabled remote cluster; failing to build the client means the metadata is inconsistent or stale, so the code deliberately crashes (fail-fast) instead of silently dropping replication.","triggerScenarios":"Raised from the Replicator's listenToClusterMetadataChange callback (service/worker/replicator/replicator.go:160) when GetRemoteAdminClient(clusterName) returns an error for a cluster present and Enabled in newClusterMetadata — e.g. the cluster was removed/degraded in config while metadata still marks it enabled, or the remote cluster's admin service address is missing/unresolvable.","commonSituations":"Misconfigured multi-cluster setup where the remote cluster entry lacks a valid admin address; race between removing a cluster from configuration and a metadata update callback; stale cluster metadata after failover; version mismatch where the peer cluster doesn't expose the admin service.","solutions":["Verify the remote cluster's admin service address is correctly configured and registered in cluster metadata before enabling it","Re-enable/re-add the remote cluster via temporal cluster add/update so metadata and client factory agree","Restart the worker after fixing config so the callback state is rebuilt from consistent metadata","Check that the peer cluster's frontend/admin service is up and resolvable via DNS/service name"],"exampleFix":"// before: server crashes when admin client cannot be created\nremoteAdminClient, err := r.clientBean.GetRemoteAdminClient(clusterName)\nif err != nil {\n    panic(fmt.Sprintf(\"Bug found in cluster metadata with error %v\", err))\n}\n// after: operator-level fix — re-register the cluster so admin address is present\n// temporal cluster update --cluster <name> --frontend-address <admin-or-frontend-host:port>\n// then restart worker: processors will be recreated with a valid client\nremoteAdminClient, err := r.clientBean.GetRemoteAdminClient(clusterName)\nif err != nil {\n    r.logger.Error(\"skipping replication processor for cluster\", tag.Cluster(clusterName), tag.Error(err))\n    return // or retry with backoff instead of crashing the process\n}","handlingStrategy":"validation","validationCode":"info, err := clusterMetadata.GetClusterInfo(clusterName)\nif err == nil && info.Enabled {\n    if info.InitializedFailoverVersion <= 0 || info.FrontendAddress == \"\" {\n        // fix cluster config before enabling replication workers\n        return fmt.Errorf(\"cluster %s enabled but address not registered\", clusterName)\n    }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always register remote clusters with `temporal cluster add/update` including a valid frontend/admin address before enabling replication","Keep cluster metadata config consistent across all clusters in the pair","Monitor metadata-change callback logs and alert on client-creation failures instead of relying on fail-fast","Restart workers only after config changes are persisted and validated"],"tags":["panic","cluster-metadata","multi-cluster-replication","grpc-client"],"backgroundTag":"cluster-metadata-missing-admin-client","analyzedSha":"bde624efd13fbd3843654058db6d9c716166318b","analyzedAt":"2026-09-01T07:18:39.080Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}