argoproj/argo-workflows · error

failed to size semaphore %s to limit %d

Error message

failed to size semaphore %s to limit %d

What it means

The limit was fetched successfully, but sem.resize(ctx, limit) returned false — the internal semaphore could not be resized to the newly discovered limit. newInternalSemaphore fails closed rather than run with a mis-sized semaphore.

Source

Thrown at workflow/sync/semaphore.go:50

	sem := &prioritySemaphore{
		name:         name,
		limitGetter:  newCachedLimit(configMapGetter, syncLimitCacheTTL),
		pending:      &priorityQueue{itemByKey: make(map[string]*item)},
		semaphore:    sema.NewWeighted(int64(0)),
		lockHolder:   make(map[string]bool),
		nextWorkflow: nextWorkflow,
		logger:       logger.get,
	}
	// Resolve the limit directly through limitGetter rather than getLimit(), since
	// getLimit() falls back to the cache's zero-value on a fetch error, which would
	// make a genuine error indistinguishable from a semaphore that legitimately
	// starts at limit 0 (e.g. an "approval gate" held closed until raised).
	limit, changed, err := sem.limitGetter.get(ctx, name)
	if err != nil {
		return nil, fmt.Errorf("failed to initialize semaphore %s: %w", name, err)
	}
	if changed && !sem.resize(ctx, limit) {
		return nil, fmt.Errorf("failed to size semaphore %s to limit %d", name, limit)
	}
	return sem, nil
}

func (s *prioritySemaphore) getLimit(ctx context.Context) int {
	limit, changed, err := s.limitGetter.get(ctx, s.name)
	if err != nil {
		// Fall back to the last known limit (returned by the cache alongside
		// the error). Returning 0 here would make release() treat a transient
		// fetch failure as a downward resize and permanently leak a slot.
		s.logger(ctx).WithError(err).WithFields(logging.Fields{
			"name":          s.name,
			"fallbackLimit": limit,
		}).Error(ctx, "failed to get limit for semaphore, using last known limit")
		return limit
	}
	if changed {
		s.resize(ctx, limit)

View on GitHub (pinned to 35bff19146)

Solutions

1. Set the semaphore limit to a sane non-negative value that is compatible with current holders, or release holds before shrinking
2. Fix the ConfigMap/DB value and re-reconcile; restart the controller if the internal state is wedged
3. Increase the limit back above current holders if you intended a temporary zero (approval gate) only for new acquisitions

Example fix

// before: shrink below holders
limit: 0   # with 2 active holders -> resize refuses
// after
currentLimit: 2
limit: 3

// validate the limit before applying it
func validLimit(v int, currentHolders int) error {
    if v < 0 { return fmt.Errorf("limit must be >= 0") }
    if v < currentHolders { return fmt.Errorf("limit %d below %d active holders", v, currentHolders) }
    return nil
}

sem, err := manager.InitializeSemaphore(ctx, name)
if err != nil && strings.Contains(err.Error(), "failed to size semaphore") {
    return fmt.Errorf("adjust semaphore limit vs current holders: %w", err)
}

When it happens

Trigger: The resize target is invalid for the semaphore's current state (e.g. shrinking the limit below the number of current holders such that resize logic refuses), during initialization when the fetched limit differs from the cached default.

Common situations: Operator lowers a semaphore limit below the count of already-held slots (e.g. limit 2 -> 0 with 2 holders); concurrent config updates racing controller initialization; bad value in the ConfigMap (negative or non-numeric parsed oddly).

Related errors

• webhook request body exceeds maximum size of 2MB
• failed to initialize semaphore %s: %w
• could not verify hold on %s for %s: %w
• hold on %s for %s is not present in the database
• invalid semaphore with both ConfigMapKeyRef and Database

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/b6b56c8e4313c7a0. Report an issue: GitHub.