{"record":{"id":"02f01cb65aa86c2c","repo":"argoproj/argo-workflows","slug":"cannot-re-establish-s-q-held-by-workflow-s-s-a","errorCode":null,"errorMessage":"cannot re-establish %s %q held by workflow %s/%s at startup: %w","messagePattern":"cannot re-establish (.+?) %q held by workflow (.+?)/(.+?) at startup: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"workflow/sync/sync_manager.go","lineNumber":321,"sourceCode":"// unresolvable holder key poisons the lock. A non-empty staleReason means the\n// hold could not be verified against its backing store (database-backed locks\n// only); the caller fails the workflow, whose teardown releases its locks.\n//\n// Poisoning, not leaving absent, is required for the init-failure case: a\n// ConfigMap-backed semaphore keeps its holders only in memory, so if we left the\n// lock absent, prepAcquire would later rebuild it with zero holders once the\n// backend recovered and let a racer acquire the slot this holder still owns. The\n// poison is lock-scoped and clears on the next controller restart.\nfunc (sm *Manager) reestablishHolder(ctx context.Context, wf *wfv1.Workflow, lockType, lockName, holder string, initLock func(context.Context, string) (semaphore, error)) (staleReason string, fatalErr error) {\n\tif sm.syncLockMap[lockName] == nil {\n\t\tlock, err := initLock(ctx, lockName)\n\t\tif err != nil {\n\t\t\tif sm.initFailureFatal(ctx, lockName) {\n\t\t\t\t// Undecodable name or a database hold with no session: we cannot\n\t\t\t\t// poison to protect the recorded hold, so halt for an operator\n\t\t\t\t// rather than risk a silent double-acquire.\n\t\t\t\tsm.log.WithField(lockType, lockName).WithError(err).Error(ctx, \"cannot initialize lock, failing closed\")\n\t\t\t\treturn \"\", fmt.Errorf(\"cannot re-establish %s %q held by workflow %s/%s at startup: %w\", lockType, lockName, wf.Namespace, wf.Name, err)\n\t\t\t}\n\t\t\t// Recoverable (e.g. transient ConfigMap unavailability) but the name\n\t\t\t// decodes, so poison protects the recorded hold without crashlooping.\n\t\t\t// Leaving the lock absent would be unsound: an in-memory semaphore\n\t\t\t// rebuilt later would have zero holders and let a racer double-acquire.\n\t\t\tsm.poison(ctx, lockName, fmt.Sprintf(\"controller could not initialize lock at startup: %v\", err))\n\t\t\treturn \"\", nil\n\t\t}\n\t\tsm.syncLockMap[lockName] = lock\n\t}\n\n\tif holder == \"\" {\n\t\treturn \"\", nil\n\t}\n\n\tkey, err := upgradeHolderKey(ctx, wf, holder, lockName)\n\tif err != nil {\n\t\tsm.poison(ctx, lockName, fmt.Sprintf(\"controller could not re-establish recorded holder %q at startup: %v\", holder, err))","sourceCodeStart":303,"sourceCodeEnd":339,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/sync/sync_manager.go#L303-L339","documentation":"At controller startup, Initialize/reestablishHolder replays recorded lock holds (semaphores/mutexes). If the recorded lock name cannot be decoded, or a database-backed hold has no live session, the hold cannot be re-established; when initFailureFatal is true the controller fails closed rather than risk a silent double-acquire. The error wraps the underlying reason (undecodable name, missing DB session).","triggerScenarios":"Controller restart with a recorded hold whose lock name fails DecodeLockName, or a database-sourced lock hold with no live session, while initFailureFatal returns true for that lock type.","commonSituations":"Sync configuration changed between restarts so recorded names no longer decode; controller running without the synchronization database while archived holds reference DB locks; corrupted or hand-edited controller state.","solutions":["Inspect the wrapped cause to distinguish undecodable name vs missing DB session","Restore the sync configuration so recorded lock names decode again","Configure the synchronization database session so DB holds can be re-established","If the stale hold is truly dead, remove the recorded hold state and restart","Only use the non-fatal (poison) path if the name decodes — do not bypass fail-closed for DB holds"],"exampleFix":"// before: syncConfig: {} (no database) -> crashloop on DB hold\n// after:\n// syncConfig:\n//   database:\n//     postgres:\n//       host: postgres\n//       database: postgres\n//       tableName: argo_sync","handlingStrategy":"validation","validationCode":"// before controller start, verify recorded lock names still decode\nif _, err := sync.DecodeLockName(context.Background(), recordedLockName); err != nil {\n    log.Fatalf(\"recorded lock %q no longer decodable: %v\", recordedLockName, err)\n}","typeGuard":"func hasDBSession(sm *sync.Manager) bool { return sm != nil && sm.DBInfo != nil && sm.DBInfo.SessionProxy != nil }","tryCatchPattern":null,"preventionTips":["Keep sync configuration stable across controller restarts","Always configure the sync database when using DB-backed semaphores","Monitor controller startup logs for 'cannot initialize lock, failing closed'","Keep sync ConfigMaps version-controlled to avoid drift"],"tags":["kubernetes","argo-workflows","sync","startup","fail-closed"],"backgroundTag":"lock-state-recovery-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}