temporalio/temporal · critical

Bug found in cluster metadata with error %v

Error message

Bug found in cluster metadata with error %v

What it means

This panic fires in the Temporal history replication replicator when a metadata-change callback sees a remote cluster marked Enabled but fails to obtain a remote admin gRPC client for it. The invariant is that cluster metadata (persisted + membership) should always contain a resolvable admin address for any enabled remote cluster; failing to build the client means the metadata is inconsistent or stale, so the code deliberately crashes (fail-fast) instead of silently dropping replication.

Source

Thrown at service/worker/replicator/replicator.go:160

		) {
			currentClusterName := r.clusterMetadata.GetCurrentClusterName()
			r.namespaceProcessorsLock.Lock()
			defer r.namespaceProcessorsLock.Unlock()
			for clusterName := range newClusterMetadata {
				if clusterName == currentClusterName {
					continue
				}
				if processor, ok := r.namespaceProcessors[clusterName]; ok {
					processor.Stop()
					delete(r.namespaceProcessors, clusterName)
				}

				if clusterInfo := newClusterMetadata[clusterName]; clusterInfo != nil && clusterInfo.Enabled {
					remoteAdminClient, err := r.clientBean.GetRemoteAdminClient(clusterName)
					if err != nil {
						// Cannot find remote cluster info.
						// This should never happen as cluster metadata should have the up-to-date data.
						panic(fmt.Sprintf("Bug found in cluster metadata with error %v", err))
					}
					processor := newReplicationMessageProcessor(
						currentClusterName,
						clusterName,
						log.With(r.logger, tag.ComponentReplicationTaskProcessor, tag.SourceCluster(clusterName)),
						r.eventLogger,
						r.emitNamespaceLifecycleEvents,
						r.eventDataProvider,
						remoteAdminClient,
						r.metricsHandler,
						r.namespaceReplicationTaskExecutor,
						r.customTaskHandler,
						r.hostInfo,
						r.serviceResolver,
						r.namespaceReplicationQueue,
						r.matchingClient,
						r.namespaceRegistry,
					)

View on GitHub (pinned to bde624efd1)

Solutions

1. Verify the remote cluster's admin service address is correctly configured and registered in cluster metadata before enabling it
2. Re-enable/re-add the remote cluster via temporal cluster add/update so metadata and client factory agree
3. Restart the worker after fixing config so the callback state is rebuilt from consistent metadata
4. Check that the peer cluster's frontend/admin service is up and resolvable via DNS/service name

Example fix

// before: server crashes when admin client cannot be created
remoteAdminClient, err := r.clientBean.GetRemoteAdminClient(clusterName)
if err != nil {
    panic(fmt.Sprintf("Bug found in cluster metadata with error %v", err))
}
// after: operator-level fix — re-register the cluster so admin address is present
// temporal cluster update --cluster <name> --frontend-address <admin-or-frontend-host:port>
// then restart worker: processors will be recreated with a valid client
remoteAdminClient, err := r.clientBean.GetRemoteAdminClient(clusterName)
if err != nil {
    r.logger.Error("skipping replication processor for cluster", tag.Cluster(clusterName), tag.Error(err))
    return // or retry with backoff instead of crashing the process
}

info, err := clusterMetadata.GetClusterInfo(clusterName)
if err == nil && info.Enabled {
    if info.InitializedFailoverVersion <= 0 || info.FrontendAddress == "" {
        // fix cluster config before enabling replication workers
        return fmt.Errorf("cluster %s enabled but address not registered", clusterName)
    }
}

When it happens

Trigger: Raised from the Replicator's listenToClusterMetadataChange callback (service/worker/replicator/replicator.go:160) when GetRemoteAdminClient(clusterName) returns an error for a cluster present and Enabled in newClusterMetadata — e.g. the cluster was removed/degraded in config while metadata still marks it enabled, or the remote cluster's admin service address is missing/unresolvable.

Common situations: Misconfigured multi-cluster setup where the remote cluster entry lacks a valid admin address; race between removing a cluster from configuration and a metadata update callback; stale cluster metadata after failover; version mismatch where the peer cluster doesn't expose the admin service.

Related errors

• Unknown cluster name: %v with given cluster initial failover
• failed to initialize current cluster metadata
• missing current cluster metadata under clusterMetadata.Clust
• current cluster %q is missing from cluster metadata
• error initializing cluster metadata manager: %w

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/9d770c8251911585. Report an issue: GitHub.