temporalio/temporal · error

error while loading metadata from cluster

Error message

error while loading metadata from cluster: %w

What it means

Wraps an error from ClusterMetadataLoader.LoadAndMergeWithStaticConfig, which loads the cluster metadata record from the DB and merges it with the static config (initial cluster name, etc.) during startup.

Solutions

1. Check the wrapped root error for the specific merge/persistence failure
2. Ensure the clusterName in config matches the existing cluster metadata record
3. Avoid simultaneous first-boot of many nodes; bootstrap once then scale
4. Verify DB write permissions for the persistence user

// ensure config cluster name matches existing record
if rec, err := getClusterMetadata(ctx); err == nil && rec.ClusterName != cfg.ClusterName {
    return fmt.Errorf("cluster name mismatch: cfg=%s db=%s", cfg.ClusterName, rec.ClusterName)
}

if err := startServer(ctx); err != nil && strings.Contains(err.Error(), "loading metadata from cluster") {
    // possible transient conflict between nodes; retry after delay
    time.Sleep(backoff)
    return startServer(ctx)
}

When it happens

Trigger: Server startup; LoadAndMergeWithStaticConfig fails, e.g. persistence update of the merged metadata fails, version conflict with a concurrent node, or invalid merged state.

Common situations: Multiple nodes starting simultaneously and racing on the metadata record version; cluster name conflicts between static config and DB record; DB write failures.

Related errors

• error initializing cluster metadata manager
• error while fetching cluster metadata
• cassandra schema version compatibility check failed
• error while updating cluster metadata
• failed to initialize current cluster metadata

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/0a4ad149f3adcd88. Report an issue: GitHub.

Appendix: source

Thrown at temporal/fx.go:767

	case *serviceerror.NotFound:
		// Initialize current cluster record
		if initErr := initCurrentClusterMetadataRecord(
			ctx,
			clusterMetadataManager,
			svc,
			indexSearchAttributes,
			logger,
		); initErr != nil {
			return svc.ClusterMetadata, svc.Persistence, initErr
		}
	default:
		return svc.ClusterMetadata, svc.Persistence, fmt.Errorf("error while fetching cluster metadata: %w", err)
	}

	clusterLoader := NewClusterMetadataLoader(clusterMetadataManager, logger)
	err = clusterLoader.LoadAndMergeWithStaticConfig(ctx, svc)
	if err != nil {
		return svc.ClusterMetadata, svc.Persistence, fmt.Errorf("error while loading metadata from cluster: %w", err)
	}
	return svc.ClusterMetadata, svc.Persistence, nil
}

func initCurrentClusterMetadataRecord(
	ctx context.Context,
	clusterMetadataManager persistence.ClusterMetadataManager,
	svc *config.Config,
	initialIndexSearchAttributes map[string]*persistencespb.IndexSearchAttributes,
	logger log.Logger,
) error {
	var clusterId string
	currentClusterName := svc.ClusterMetadata.CurrentClusterName
	currentClusterInfo := svc.ClusterMetadata.ClusterInformation[currentClusterName]
	if uuid.Validate(currentClusterInfo.ClusterID) != nil {
		if currentClusterInfo.ClusterID != "" {
			logger.Warn("Cluster Id in Cluster Metadata config is not a valid uuid. Generating a new Cluster Id")
		}

View on GitHub (pinned to bde624efd1)