What causes "could not verify hold on %s for %s: %w" in argoproj/argo-workflows?

Check the underlying wrapped error and restore database connectivity / credentials, then retry the workflow

could not verify hold on %s for %s: %w

Error message

could not verify hold on %s for %s: %w

What it means

This error is returned by databaseSemaphore.reacquire when it cannot query the current holders of a database-backed semaphore from the database after the controller restarts. The wrap preserves the underlying DB error (connection failure, query error, table missing, etc.). The controller deliberately fails rather than guess: it cannot distinguish a lost hold from a stale one without reading holders.

Source

Thrown at workflow/sync/database_semaphore.go:393

		"reason":          "limit exceeded",
		"current_holders": len(existing),
		"limit":           limit,
	}).Info(ctx, "Acquire failed")
	return false, nil
}

// reacquire asserts at startup that the recorded holder still holds this lock
// in the database. The database is the single source of truth for a
// database-backed lock: the held row is durable and survives the controller
// restart, so nothing is inserted or mutated here. A missing row means the
// hold no longer exists - e.g. it was expired by ExpireInactiveLocks while the
// controller was down and may since have been acquired by another holder - so
// the workflow's recorded hold is stale and the caller fails the workflow
// rather than resurrect a hold the database does not back.
func (s *databaseSemaphore) reacquire(ctx context.Context, holderKey string, tx *sqldb.SessionProxy) error {
	holders, err := s.currentHoldersSession(ctx, tx)
	if err != nil {
		return fmt.Errorf("could not verify hold on %s for %s: %w", s.longDBKey(), holderKey, err)
	}
	if !slices.Contains(holders, holderKey) {
		return fmt.Errorf("hold on %s for %s is not present in the database", s.longDBKey(), holderKey)
	}
	return nil
}

func (s *databaseSemaphore) tryAcquire(ctx context.Context, holderKey string, tx *sqldb.SessionProxy) (bool, string, error) {
	logger := s.logger(ctx)
	acq, already, msg := s.checkAcquire(ctx, holderKey, tx)
	if already {
		logger.WithFields(logging.Fields{
			"key":     holderKey,
			"result":  true,
			"message": msg,
		}).Info(ctx, "tryAcquire - already held")
		return true, msg, nil
	}

View on GitHub (pinned to 35bff19146)

Solutions

Check the underlying wrapped error and restore database connectivity / credentials, then retry the workflow
Verify the database-backed sync tables exist and migrations ran (upgrade the controller so schema is initialized)
If the DB is healthy, inspect controller logs for pool exhaustion and increase connection limits or restart the controller
As a last resort, re-submit the workflow so locks are re-established from scratch

Example fix

// before: reacquire fails on transient DB error
return fmt.Errorf("could not verify hold on %s for %s: %w", s.longDBKey(), holderKey, err)
// after: operator fixes DB access, e.g. correct DSN in controller config
config.Database:
  postgres:
    host: postgres.default.svc
    database: argo   # credentials must match the secret referenced by the controller

Defensive patterns

Strategy: retry

Validate before calling

// pre-check DB reachability before workflows with DB semaphores run
import "database/sql"
func dbReachable(dsn string) error {
    db, err := sql.Open("postgres", dsn)
    if err != nil { return err }
    defer db.Close()
    return db.Ping()
}

Try / catch

err := wf.Retry(ctx)
if err != nil && strings.Contains(err.Error(), "could not verify hold on") {
    // underlying DB error is wrapped; check connectivity then retry
    if dbReachable(dsn) == nil { _ = wf.Retry(ctx) }
}

Prevention

Monitor database health/latency from the controller
Use connection pooling with sane limits and retries
Run DB migrations as part of upgrades before workloads depend on DB syncs

When it happens

Trigger: A workflow holding a database semaphore lock is reconciled after a controller restart; reacquire calls currentHoldersSession inside the tx and the SELECT of holder rows fails (DB down, connectivity blip, timeout, bad DSN, migrations not applied).

Common situations: Postgres/MySQL outage or network partition during controller startup; wrong database credentials in the semaphore config; workflow archive/lock tables not created; transient connection-pool exhaustion under load.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/26e39254f7d01e23. Report an issue: GitHub.