{"record":{"id":"6a2cbef3ebaca394","repo":"argoproj/argo-workflows","slug":"hold-on-s-for-s-is-not-present-in-the-database","errorCode":null,"errorMessage":"hold on %s for %s is not present in the database","messagePattern":"hold on (.+?) for (.+?) is not present in the database","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/sync/database_semaphore.go","lineNumber":396,"sourceCode":"\t}).Info(ctx, \"Acquire failed\")\n\treturn false, nil\n}\n\n// reacquire asserts at startup that the recorded holder still holds this lock\n// in the database. The database is the single source of truth for a\n// database-backed lock: the held row is durable and survives the controller\n// restart, so nothing is inserted or mutated here. A missing row means the\n// hold no longer exists - e.g. it was expired by ExpireInactiveLocks while the\n// controller was down and may since have been acquired by another holder - so\n// the workflow's recorded hold is stale and the caller fails the workflow\n// rather than resurrect a hold the database does not back.\nfunc (s *databaseSemaphore) reacquire(ctx context.Context, holderKey string, tx *sqldb.SessionProxy) error {\n\tholders, err := s.currentHoldersSession(ctx, tx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"could not verify hold on %s for %s: %w\", s.longDBKey(), holderKey, err)\n\t}\n\tif !slices.Contains(holders, holderKey) {\n\t\treturn fmt.Errorf(\"hold on %s for %s is not present in the database\", s.longDBKey(), holderKey)\n\t}\n\treturn nil\n}\n\nfunc (s *databaseSemaphore) tryAcquire(ctx context.Context, holderKey string, tx *sqldb.SessionProxy) (bool, string, error) {\n\tlogger := s.logger(ctx)\n\tacq, already, msg := s.checkAcquire(ctx, holderKey, tx)\n\tif already {\n\t\tlogger.WithFields(logging.Fields{\n\t\t\t\"key\":     holderKey,\n\t\t\t\"result\":  true,\n\t\t\t\"message\": msg,\n\t\t}).Info(ctx, \"tryAcquire - already held\")\n\t\treturn true, msg, nil\n\t}\n\tif !acq {\n\t\tlogger.WithFields(logging.Fields{\n\t\t\t\"key\":     holderKey,","sourceCodeStart":378,"sourceCodeEnd":414,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/sync/database_semaphore.go#L378-L414","documentation":"reacquire successfully read the holder list from the database, but the workflow's holderKey is not among them. After a controller outage the recorded hold is stale — another holder may have taken or released the slot — so the controller fails the workflow instead of resurrecting a hold the database does not back.","triggerScenarios":"Controller was down while the semaphore was modified; on restart, reconciliation calls reacquire(holderKey) and slices.Contains(holders, holderKey) is false — e.g. the hold expired/was deleted, another workflow took the slot, or the holder rows were cleaned up.","commonSituations":"Long controller downtime with concurrent workflow changes; operator manually deleting lock rows; semaphore limit raised/lowered during the outage so holds were reassigned; database-backed sync rows purged by retention jobs.","solutions":["Re-submit or retry the workflow so it competes for the semaphore again under current holder state","Inspect the holders in the sync database table for the lock and reconcile with the workflow's recorded hold","Check whether another workflow legitimately acquired the slot during downtime and either wait or increase the semaphore limit","Prevent long outages by running HA replicas of the controller so reacquire happens promptly"],"exampleFix":"// before: stale hold after downtime -> workflow failed\n// operator increases limit so pending workflows can acquire:\n// ConfigMap semaphore config\nsynchronization:\n  semaphore:\n    database: {key: my-lock}  # adjust limit/rows in DB to re-admit\n// after: retry the workflow\nargo retry failed-wf-1234","handlingStrategy":"fallback","validationCode":"// before relying on a DB-held lock across restarts, confirm the hold exists\nrows, err := db.Query(\"SELECT holder FROM sync_holders WHERE lock = $1\", lockKey)\nif err != nil { return err }\nfound := false\nfor rows.Next() {\n    var h string; _ = rows.Scan(&h)\n    if h == holderKey { found = true }\n}\nif !found { /* re-acquire instead of assuming hold persists */ }","typeGuard":null,"tryCatchPattern":"err := runWorkflow(ctx)\nif err != nil && strings.Contains(err.Error(), \"is not present in the database\") {\n    // hold is stale: re-submit/retry so the lock is re-acquired cleanly\n    _ = retryWorkflow(ctx, wfName)\n}","preventionTips":["Avoid long single-replica controller outages; run HA controller replicas","Don't manually delete sync holder rows while workflows are pending","Document that DB-backed holds do not survive controller downtime gaps"],"tags":["semaphore","database","stale-state"],"backgroundTag":"stale-lock-hold","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}