{"record":{"id":"26e39254f7d01e23","repo":"argoproj/argo-workflows","slug":"could-not-verify-hold-on-s-for-s-w","errorCode":null,"errorMessage":"could not verify hold on %s for %s: %w","messagePattern":"could not verify hold on (.+?) for (.+?): %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/sync/database_semaphore.go","lineNumber":393,"sourceCode":"\t\t\"reason\":          \"limit exceeded\",\n\t\t\"current_holders\": len(existing),\n\t\t\"limit\":           limit,\n\t}).Info(ctx, \"Acquire failed\")\n\treturn false, nil\n}\n\n// reacquire asserts at startup that the recorded holder still holds this lock\n// in the database. The database is the single source of truth for a\n// database-backed lock: the held row is durable and survives the controller\n// restart, so nothing is inserted or mutated here. A missing row means the\n// hold no longer exists - e.g. it was expired by ExpireInactiveLocks while the\n// controller was down and may since have been acquired by another holder - so\n// the workflow's recorded hold is stale and the caller fails the workflow\n// rather than resurrect a hold the database does not back.\nfunc (s *databaseSemaphore) reacquire(ctx context.Context, holderKey string, tx *sqldb.SessionProxy) error {\n\tholders, err := s.currentHoldersSession(ctx, tx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"could not verify hold on %s for %s: %w\", s.longDBKey(), holderKey, err)\n\t}\n\tif !slices.Contains(holders, holderKey) {\n\t\treturn fmt.Errorf(\"hold on %s for %s is not present in the database\", s.longDBKey(), holderKey)\n\t}\n\treturn nil\n}\n\nfunc (s *databaseSemaphore) tryAcquire(ctx context.Context, holderKey string, tx *sqldb.SessionProxy) (bool, string, error) {\n\tlogger := s.logger(ctx)\n\tacq, already, msg := s.checkAcquire(ctx, holderKey, tx)\n\tif already {\n\t\tlogger.WithFields(logging.Fields{\n\t\t\t\"key\":     holderKey,\n\t\t\t\"result\":  true,\n\t\t\t\"message\": msg,\n\t\t}).Info(ctx, \"tryAcquire - already held\")\n\t\treturn true, msg, nil\n\t}","sourceCodeStart":375,"sourceCodeEnd":411,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/sync/database_semaphore.go#L375-L411","documentation":"This error is returned by databaseSemaphore.reacquire when it cannot query the current holders of a database-backed semaphore from the database after the controller restarts. The wrap preserves the underlying DB error (connection failure, query error, table missing, etc.). The controller deliberately fails rather than guess: it cannot distinguish a lost hold from a stale one without reading holders.","triggerScenarios":"A workflow holding a database semaphore lock is reconciled after a controller restart; reacquire calls currentHoldersSession inside the tx and the SELECT of holder rows fails (DB down, connectivity blip, timeout, bad DSN, migrations not applied).","commonSituations":"Postgres/MySQL outage or network partition during controller startup; wrong database credentials in the semaphore config; workflow archive/lock tables not created; transient connection-pool exhaustion under load.","solutions":["Check the underlying wrapped error and restore database connectivity / credentials, then retry the workflow","Verify the database-backed sync tables exist and migrations ran (upgrade the controller so schema is initialized)","If the DB is healthy, inspect controller logs for pool exhaustion and increase connection limits or restart the controller","As a last resort, re-submit the workflow so locks are re-established from scratch"],"exampleFix":"// before: reacquire fails on transient DB error\nreturn fmt.Errorf(\"could not verify hold on %s for %s: %w\", s.longDBKey(), holderKey, err)\n// after: operator fixes DB access, e.g. correct DSN in controller config\nconfig.Database:\n  postgres:\n    host: postgres.default.svc\n    database: argo   # credentials must match the secret referenced by the controller","handlingStrategy":"retry","validationCode":"// pre-check DB reachability before workflows with DB semaphores run\nimport \"database/sql\"\nfunc dbReachable(dsn string) error {\n    db, err := sql.Open(\"postgres\", dsn)\n    if err != nil { return err }\n    defer db.Close()\n    return db.Ping()\n}","typeGuard":null,"tryCatchPattern":"err := wf.Retry(ctx)\nif err != nil && strings.Contains(err.Error(), \"could not verify hold on\") {\n    // underlying DB error is wrapped; check connectivity then retry\n    if dbReachable(dsn) == nil { _ = wf.Retry(ctx) }\n}","preventionTips":["Monitor database health/latency from the controller","Use connection pooling with sane limits and retries","Run DB migrations as part of upgrades before workloads depend on DB syncs"],"tags":["database","semaphore","controller-restart"],"backgroundTag":"database-connectivity-failure","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}