argoproj/argo-workflows · critical

cannot re-establish %s %q held by workflow %s/%s at startup:

Error message

cannot re-establish %s %q held by workflow %s/%s at startup: %w

What it means

At controller startup, Initialize/reestablishHolder replays recorded lock holds (semaphores/mutexes). If the recorded lock name cannot be decoded, or a database-backed hold has no live session, the hold cannot be re-established; when initFailureFatal is true the controller fails closed rather than risk a silent double-acquire. The error wraps the underlying reason (undecodable name, missing DB session).

Source

Thrown at workflow/sync/sync_manager.go:321

// unresolvable holder key poisons the lock. A non-empty staleReason means the
// hold could not be verified against its backing store (database-backed locks
// only); the caller fails the workflow, whose teardown releases its locks.
//
// Poisoning, not leaving absent, is required for the init-failure case: a
// ConfigMap-backed semaphore keeps its holders only in memory, so if we left the
// lock absent, prepAcquire would later rebuild it with zero holders once the
// backend recovered and let a racer acquire the slot this holder still owns. The
// poison is lock-scoped and clears on the next controller restart.
func (sm *Manager) reestablishHolder(ctx context.Context, wf *wfv1.Workflow, lockType, lockName, holder string, initLock func(context.Context, string) (semaphore, error)) (staleReason string, fatalErr error) {
	if sm.syncLockMap[lockName] == nil {
		lock, err := initLock(ctx, lockName)
		if err != nil {
			if sm.initFailureFatal(ctx, lockName) {
				// Undecodable name or a database hold with no session: we cannot
				// poison to protect the recorded hold, so halt for an operator
				// rather than risk a silent double-acquire.
				sm.log.WithField(lockType, lockName).WithError(err).Error(ctx, "cannot initialize lock, failing closed")
				return "", fmt.Errorf("cannot re-establish %s %q held by workflow %s/%s at startup: %w", lockType, lockName, wf.Namespace, wf.Name, err)
			}
			// Recoverable (e.g. transient ConfigMap unavailability) but the name
			// decodes, so poison protects the recorded hold without crashlooping.
			// Leaving the lock absent would be unsound: an in-memory semaphore
			// rebuilt later would have zero holders and let a racer double-acquire.
			sm.poison(ctx, lockName, fmt.Sprintf("controller could not initialize lock at startup: %v", err))
			return "", nil
		}
		sm.syncLockMap[lockName] = lock
	}

	if holder == "" {
		return "", nil
	}

	key, err := upgradeHolderKey(ctx, wf, holder, lockName)
	if err != nil {
		sm.poison(ctx, lockName, fmt.Sprintf("controller could not re-establish recorded holder %q at startup: %v", holder, err))

View on GitHub (pinned to 35bff19146)

Solutions

1. Inspect the wrapped cause to distinguish undecodable name vs missing DB session
2. Restore the sync configuration so recorded lock names decode again
3. Configure the synchronization database session so DB holds can be re-established
4. If the stale hold is truly dead, remove the recorded hold state and restart
5. Only use the non-fatal (poison) path if the name decodes — do not bypass fail-closed for DB holds

Example fix

// before: syncConfig: {} (no database) -> crashloop on DB hold
// after:
// syncConfig:
//   database:
//     postgres:
//       host: postgres
//       database: postgres
//       tableName: argo_sync

// before controller start, verify recorded lock names still decode
if _, err := sync.DecodeLockName(context.Background(), recordedLockName); err != nil {
    log.Fatalf("recorded lock %q no longer decodable: %v", recordedLockName, err)
}

func hasDBSession(sm *sync.Manager) bool { return sm != nil && sm.DBInfo != nil && sm.DBInfo.SessionProxy != nil }

When it happens

Trigger: Controller restart with a recorded hold whose lock name fails DecodeLockName, or a database-sourced lock hold with no live session, while initFailureFatal returns true for that lock type.

Common situations: Sync configuration changed between restarts so recorded names no longer decode; controller running without the synchronization database while archived holds reference DB locks; corrupted or hand-edited controller state.

Related errors

• Artifact driver validation failed: The following artifact dr
• failed to read container args file %s: %w
• failed to unmarshal container args: %w
• failed to read template: %w
• failed to start command: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/02f01cb65aa86c2c. Report an issue: GitHub.