What causes "failed to initialize semaphore %s: %w" in argoproj/argo-workflows?

Check controller database connectivity and persistence config (host, credentials, schema)

failed to initialize semaphore %s: %w

Error message

failed to initialize semaphore %s: %w

What it means

newDatabaseSemaphore (database-backed semaphore/mutex, sync package) eagerly verifies it can read the semaphore's limit from the database via limitGetter.get at creation time. On error, it wraps the failure with the semaphore name so callers know which semaphore could not be initialized; creation is aborted rather than silently starting with a cached zero limit (which would make a real error look like a legitimately closed semaphore).

Source

Thrown at workflow/sync/database_semaphore.go:48

		lockType: lockTypeSemaphore,
	}
	sem := &databaseSemaphore{
		name:         name,
		shortDBKey:   dbKey,
		limitGetter:  nil,
		nextWorkflow: nextWorkflow,
		logger:       logger.get,
		info:         info,
		queries:      syncdb.NewSyncQueries(info.SessionProxy, info.Config),
		isMutex:      false,
	}
	sem.limitGetter = newCachedLimit(sem.getLimitFromDB, syncLimitCacheTTL)
	// Resolve the limit directly through limitGetter rather than getLimit(), since
	// getLimit() falls back to the cache's zero-value on a fetch error, which would
	// make a genuine error indistinguishable from a semaphore that legitimately
	// starts at limit 0 (e.g. an "approval gate" held closed until raised).
	if _, _, err := sem.limitGetter.get(ctx, dbKey); err != nil {
		return nil, fmt.Errorf("failed to initialize semaphore %s: %w", name, err)
	}
	return sem, nil
}

func (s *databaseSemaphore) longDBKey() string {
	if s.isMutex {
		return "mtx/" + s.shortDBKey
	}
	return "sem/" + s.shortDBKey
}

func (s *databaseSemaphore) getLimitFromDB(ctx context.Context, _ string) (int, error) {
	logger := s.logger(ctx)
	// Update the limit from the database
	limit, err := s.queries.GetSemaphoreLimit(ctx, s.shortDBKey)
	if err != nil {
		logger.WithField("key", s.shortDBKey).WithError(err).Error(ctx, "Failed to get limit")
		return 0, err

View on GitHub (pinned to 35bff19146)

Solutions

Check controller database connectivity and persistence config (host, credentials, schema)
Verify the semaphore limit row/config exists in the database and the sync configmap/database is initialized
Check controller logs for the wrapped underlying error (%w) to identify the root cause (timeout, refused connection, SQL syntax)
Retry after DB recovery — semaphore creation is transient-failure prone; restart the controller if semaphore state is stale
Ensure the sync DB migrations ran (make the semaphore tables match your Argo version)

Example fix

# before: controller config pointing at unreachable DB
persistence:
  postgresql:
    host: wrong-host
# after: corrected DB endpoint
persistence:
  postgresql:
    host: postgres.default.svc.cluster.local
    database: argo

Defensive patterns

Strategy: retry

Validate before calling

// before creating workflows using DB semaphores, verify the controller can reach the DB
// kubectl exec deploy/workflow-controller -- env | grep DB; check readiness of the DB service

Try / catch

sem, err := initializeSemaphore(ctx, name, ...)
if err != nil && strings.Contains(err.Error(), "failed to initialize semaphore") {
	// check DB connectivity, then retry with backoff
	err = retry.Do(ctx, func() error { return retryableInit(ctx, name) })
}

Prevention

Monitor DB (Postgres/MySQL) availability and controller-to-DB connectivity
Verify persistence/sync config (credentials, host, schema) after any config change
Run sync DB migrations when upgrading Argo versions
Ensure semaphore limits are seeded in the sync database before workflows reference them

When it happens

Trigger: initializeSemaphore creating a database-backed semaphore when the backing SQL query fails — DB unreachable, semaphore row/config missing, table schema mismatch, connection timeout, or bad database credentials in the controller's persistence config.

Common situations: Postgres/MySQL archive DB down or misconfigured (configmap persistence settings); semaphore defined in workflow spec but its DB record never seeded; network partition between controller and DB; schema migrations out of sync after upgrading Argo.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/73ce7f4ac6d5fe10. Report an issue: GitHub.