argoproj/argo-workflows · error
was unable to determine level for %s
Error message
was unable to determine level for %s
What it means
getWorkflowSyncLevelByName walks the workflow's synchronization references (workflow-level, template-level, template default) to determine at which level a lock with the given name was declared. If no level matches and no earlier error was recorded, it returns ErrorLevel with 'was unable to determine level for <lockName>'.
Source
Thrown at workflow/sync/sync_manager.go:266
if template.Synchronization != nil {
syncItems, err := allSyncItems(template.Synchronization)
if err != nil {
return ErrorLevel, err
}
for _, syncItem := range syncItems {
syncLockName, err := syncItem.lockName(wf.Namespace)
if err != nil {
lastErr = err
continue
}
if lockName == syncLockName.String(ctx) {
return TemplateLevel, nil
}
}
}
}
if lastErr == nil {
lastErr = fmt.Errorf("was unable to determine level for %s", lockName)
}
return ErrorLevel, lastErr
}
// initFailureFatal reports whether a failure to (re)establish a lock at startup
// is unrecoverable and must fail closed (crashloop). Only two cases qualify:
// - the lock name is undecodable, so there is no key to poison under and no way
// to prove the workflow's spec re-acquires the same lock; and
// - the lock is database-backed but no database session is configured, so
// nothing can back the lock.
//
// Everything else - a transient ConfigMap/DB read failure, a limit fetch
// returning 0 - is recoverable: the name decodes, so the lock can be poisoned
// (the poison key matches what a racer would compute) without halting the whole
// controller.
func (sm *Manager) initFailureFatal(ctx context.Context, lockName string) bool {
decoded, err := DecodeLockName(ctx, lockName)
if err != nil {View on GitHub (pinned to 35bff19146)
Solutions
- Verify the lock name in the error matches a semaphore/mutex declared in the workflow spec; fix renames or revert the spec change
- Retry/re-submit the workflow to rebuild holder state consistent with the current spec
- Clean stale node/holder status entries that reference the old lock name
Example fix
// before: renamed lock with stale holders
synchronization:
semaphore:
configMapKeyRef: {name: cm, key: new-key}
// after: keep the key stable across edits, or re-submit
synchronization:
semaphore:
configMapKeyRef: {name: cm, key: original-key} Defensive patterns
Strategy: try-catch
Validate before calling
// verify the lock name appears in the workflow spec before relying on holds
func declaresLock(wf *Workflow, lockName string) bool {
return strings.Contains(fmt.Sprintf("%v", wf.Spec.Synchronization), filepath.Base(lockName))
} Try / catch
level, err := sm.GetWorkflowSyncLevelByName(ctx, wf, lockName)
if err != nil && strings.Contains(err.Error(), "unable to determine level") {
return fmt.Errorf("lock %q not declared in workflow spec; fix spec or rebuild state: %w", lockName, err)
} Prevention
- Keep semaphore/mutex names stable across workflow edits, or re-submit instead of editing in place
- Use `argo lint` after renaming locks to catch stale references
- Purge stale node status when removing synchronization from templates
When it happens
Trigger: upgradeHolderKey asks for the sync level of a holder's lock during controller reconciliation; the lockName string doesn't match any semaphore/mutex declared anywhere in the workflow (renamed lock, deleted node, stale holder entry).
Common situations: Workflow spec edited to rename a semaphore/mutex while old holds persisted; node status referencing a lock from a removed template; decoding a lock name that never belonged to the workflow.
AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03).
Data as JSON: /api/errors/7352fb31364ddede.
Report an issue: GitHub.