argoproj/argo-workflows · error

holderkey is empty

Error message

holderkey is empty

What it means

Manager.getWorkflowKey parses a holder key of the form '<namespace>/<workflow>/<node>' down to '<namespace>/<workflow>'. An empty holder key cannot identify any workflow, so this error is returned and used by CheckWorkflowExistence to decide whether the holder's workflow still exists.

Source

Thrown at workflow/sync/sync_manager.go:110

		syncLimitCacheTTL: syncLimitCacheTTL,
		workflowExists:    workflowExists,
		dbInfo:            dbInfo,
		queries:           syncdb.NewSyncQueries(sessionProxy, dbInfo.Config),
		log:               log,
	}
	log.WithField("dbConfigured", sm.dbInfo.SessionProxy != nil).Info(ctx, "Sync manager initialized")
	sm.dbInfo.Migrate(ctx)

	if sm.dbInfo.SessionProxy != nil {
		sm.backgroundNotifier(ctx, config.PollSeconds)
		sm.dbControllerHeartbeat(ctx, config.HeartbeatSeconds)
	}
	return sm
}

func (sm *Manager) getWorkflowKey(key string) (string, error) {
	if key == "" {
		return "", fmt.Errorf("holderkey is empty")
	}
	items := strings.Split(key, "/")
	if len(items) < 2 {
		return "", fmt.Errorf("invalid holderkey format")
	}
	return fmt.Sprintf("%s/%s", items[0], items[1]), nil
}

func (sm *Manager) CheckWorkflowExistence(ctx context.Context) {
	defer runtimeutil.HandleCrashWithContext(ctx, runtimeutil.PanicHandlers...)

	sm.lock.Lock()
	defer sm.lock.Unlock()

	sm.log.Debug(ctx, "Check the workflow existence")
	for _, lock := range sm.syncLockMap {
		holders, err := lock.getCurrentHolders(ctx)
		if err != nil {

View on GitHub (pinned to 35bff19146)

Solutions

1. Inspect the workflow's synchronization status for empty holder entries and clean them up (re-submit or patch the workflow)
2. Check the sync database/ConfigMap state for blank holder keys and remove them
3. If reproducible, audit the code path that acquires the lock to ensure holderKey is always set before registration

Example fix

// before: holder registered with empty key
holderKey := ""
sm.tryAcquire(lockName, holderKey, ...)
// after
holderKey := fmt.Sprintf("%s/%s/%s", wf.Namespace, wf.Name, nodeName)
sm.tryAcquire(lockName, holderKey, ...)

func validateHolderKey(key string) error {
    if key == "" { return fmt.Errorf("holderkey is empty") }
    if strings.Count(key, "/") < 2 { return fmt.Errorf("holderkey must be namespace/workflow/node") }
    return nil
}

func isHolderKey(s string) bool {
    parts := strings.Split(s, "/")
    return len(parts) >= 2 && parts[0] != "" && parts[1] != ""
}

err := acquire(ctx, lockName, holderKey)
if err != nil && strings.Contains(err.Error(), "holderkey is empty") {
    return fmt.Errorf("holder key must be computed before acquiring: %w", err)
}

When it happens

Trigger: CheckWorkflowExistence iterates holders of a lock and passes an empty string as the key — typically a holder entry recorded with no holderKey (corrupted sync state, or a semaphore/mutex acquired with an unset key).

Common situations: Status/sync maps corrupted after a crash or manual edit of workflow status; code paths creating holds before the holder key is computed; database rows with empty holder columns.

Related errors

• invalid holderkey format
• failed to initialize semaphore %s: %w
• could not verify hold on %s for %s: %w
• hold on %s for %s is not present in the database
• invalid semaphore with both ConfigMapKeyRef and Database

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/16bde295fc70b073. Report an issue: GitHub.