argoproj/argo-workflows · error

hold on %s for %s is not present in the database

Error message

hold on %s for %s is not present in the database

What it means

reacquire successfully read the holder list from the database, but the workflow's holderKey is not among them. After a controller outage the recorded hold is stale — another holder may have taken or released the slot — so the controller fails the workflow instead of resurrecting a hold the database does not back.

Source

Thrown at workflow/sync/database_semaphore.go:396

	}).Info(ctx, "Acquire failed")
	return false, nil
}

// reacquire asserts at startup that the recorded holder still holds this lock
// in the database. The database is the single source of truth for a
// database-backed lock: the held row is durable and survives the controller
// restart, so nothing is inserted or mutated here. A missing row means the
// hold no longer exists - e.g. it was expired by ExpireInactiveLocks while the
// controller was down and may since have been acquired by another holder - so
// the workflow's recorded hold is stale and the caller fails the workflow
// rather than resurrect a hold the database does not back.
func (s *databaseSemaphore) reacquire(ctx context.Context, holderKey string, tx *sqldb.SessionProxy) error {
	holders, err := s.currentHoldersSession(ctx, tx)
	if err != nil {
		return fmt.Errorf("could not verify hold on %s for %s: %w", s.longDBKey(), holderKey, err)
	}
	if !slices.Contains(holders, holderKey) {
		return fmt.Errorf("hold on %s for %s is not present in the database", s.longDBKey(), holderKey)
	}
	return nil
}

func (s *databaseSemaphore) tryAcquire(ctx context.Context, holderKey string, tx *sqldb.SessionProxy) (bool, string, error) {
	logger := s.logger(ctx)
	acq, already, msg := s.checkAcquire(ctx, holderKey, tx)
	if already {
		logger.WithFields(logging.Fields{
			"key":     holderKey,
			"result":  true,
			"message": msg,
		}).Info(ctx, "tryAcquire - already held")
		return true, msg, nil
	}
	if !acq {
		logger.WithFields(logging.Fields{
			"key":     holderKey,

View on GitHub (pinned to 35bff19146)

Solutions

1. Re-submit or retry the workflow so it competes for the semaphore again under current holder state
2. Inspect the holders in the sync database table for the lock and reconcile with the workflow's recorded hold
3. Check whether another workflow legitimately acquired the slot during downtime and either wait or increase the semaphore limit
4. Prevent long outages by running HA replicas of the controller so reacquire happens promptly

Example fix

// before: stale hold after downtime -> workflow failed
// operator increases limit so pending workflows can acquire:
// ConfigMap semaphore config
synchronization:
  semaphore:
    database: {key: my-lock}  # adjust limit/rows in DB to re-admit
// after: retry the workflow
argo retry failed-wf-1234

// before relying on a DB-held lock across restarts, confirm the hold exists
rows, err := db.Query("SELECT holder FROM sync_holders WHERE lock = $1", lockKey)
if err != nil { return err }
found := false
for rows.Next() {
    var h string; _ = rows.Scan(&h)
    if h == holderKey { found = true }
}
if !found { /* re-acquire instead of assuming hold persists */ }

err := runWorkflow(ctx)
if err != nil && strings.Contains(err.Error(), "is not present in the database") {
    // hold is stale: re-submit/retry so the lock is re-acquired cleanly
    _ = retryWorkflow(ctx, wfName)
}

When it happens

Trigger: Controller was down while the semaphore was modified; on restart, reconciliation calls reacquire(holderKey) and slices.Contains(holders, holderKey) is false — e.g. the hold expired/was deleted, another workflow took the slot, or the holder rows were cleaned up.

Common situations: Long controller downtime with concurrent workflow changes; operator manually deleting lock rows; semaphore limit raised/lowered during the outage so holds were reassigned; database-backed sync rows purged by retention jobs.

Related errors

• failed to initialize semaphore %s: %w
• could not verify hold on %s for %s: %w
• database session is not available for semaphore %s
• operation %v is not supported
• offload node status is not supported

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/6a2cbef3ebaca394. Report an issue: GitHub.