{"record":{"id":"73ce7f4ac6d5fe10","repo":"argoproj/argo-workflows","slug":"failed-to-initialize-semaphore-s-w","errorCode":null,"errorMessage":"failed to initialize semaphore %s: %w","messagePattern":"failed to initialize semaphore (.+?): %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/sync/database_semaphore.go","lineNumber":48,"sourceCode":"\t\tlockType: lockTypeSemaphore,\n\t}\n\tsem := &databaseSemaphore{\n\t\tname:         name,\n\t\tshortDBKey:   dbKey,\n\t\tlimitGetter:  nil,\n\t\tnextWorkflow: nextWorkflow,\n\t\tlogger:       logger.get,\n\t\tinfo:         info,\n\t\tqueries:      syncdb.NewSyncQueries(info.SessionProxy, info.Config),\n\t\tisMutex:      false,\n\t}\n\tsem.limitGetter = newCachedLimit(sem.getLimitFromDB, syncLimitCacheTTL)\n\t// Resolve the limit directly through limitGetter rather than getLimit(), since\n\t// getLimit() falls back to the cache's zero-value on a fetch error, which would\n\t// make a genuine error indistinguishable from a semaphore that legitimately\n\t// starts at limit 0 (e.g. an \"approval gate\" held closed until raised).\n\tif _, _, err := sem.limitGetter.get(ctx, dbKey); err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to initialize semaphore %s: %w\", name, err)\n\t}\n\treturn sem, nil\n}\n\nfunc (s *databaseSemaphore) longDBKey() string {\n\tif s.isMutex {\n\t\treturn \"mtx/\" + s.shortDBKey\n\t}\n\treturn \"sem/\" + s.shortDBKey\n}\n\nfunc (s *databaseSemaphore) getLimitFromDB(ctx context.Context, _ string) (int, error) {\n\tlogger := s.logger(ctx)\n\t// Update the limit from the database\n\tlimit, err := s.queries.GetSemaphoreLimit(ctx, s.shortDBKey)\n\tif err != nil {\n\t\tlogger.WithField(\"key\", s.shortDBKey).WithError(err).Error(ctx, \"Failed to get limit\")\n\t\treturn 0, err","sourceCodeStart":30,"sourceCodeEnd":66,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/sync/database_semaphore.go#L30-L66","documentation":"newDatabaseSemaphore (database-backed semaphore/mutex, sync package) eagerly verifies it can read the semaphore's limit from the database via limitGetter.get at creation time. On error, it wraps the failure with the semaphore name so callers know which semaphore could not be initialized; creation is aborted rather than silently starting with a cached zero limit (which would make a real error look like a legitimately closed semaphore).","triggerScenarios":"initializeSemaphore creating a database-backed semaphore when the backing SQL query fails — DB unreachable, semaphore row/config missing, table schema mismatch, connection timeout, or bad database credentials in the controller's persistence config.","commonSituations":"Postgres/MySQL archive DB down or misconfigured (configmap persistence settings); semaphore defined in workflow spec but its DB record never seeded; network partition between controller and DB; schema migrations out of sync after upgrading Argo.","solutions":["Check controller database connectivity and persistence config (host, credentials, schema)","Verify the semaphore limit row/config exists in the database and the sync configmap/database is initialized","Check controller logs for the wrapped underlying error (%w) to identify the root cause (timeout, refused connection, SQL syntax)","Retry after DB recovery — semaphore creation is transient-failure prone; restart the controller if semaphore state is stale","Ensure the sync DB migrations ran (make the semaphore tables match your Argo version)"],"exampleFix":"# before: controller config pointing at unreachable DB\npersistence:\n  postgresql:\n    host: wrong-host\n# after: corrected DB endpoint\npersistence:\n  postgresql:\n    host: postgres.default.svc.cluster.local\n    database: argo","handlingStrategy":"retry","validationCode":"// before creating workflows using DB semaphores, verify the controller can reach the DB\n// kubectl exec deploy/workflow-controller -- env | grep DB; check readiness of the DB service","typeGuard":null,"tryCatchPattern":"sem, err := initializeSemaphore(ctx, name, ...)\nif err != nil && strings.Contains(err.Error(), \"failed to initialize semaphore\") {\n\t// check DB connectivity, then retry with backoff\n\terr = retry.Do(ctx, func() error { return retryableInit(ctx, name) })\n}","preventionTips":["Monitor DB (Postgres/MySQL) availability and controller-to-DB connectivity","Verify persistence/sync config (credentials, host, schema) after any config change","Run sync DB migrations when upgrading Argo versions","Ensure semaphore limits are seeded in the sync database before workflows reference them"],"tags":["sync","semaphore","database","persistence","workflow-controller"],"backgroundTag":"semaphore-init-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}