argoproj/argo-workflows · critical

failed to create initial database session: %w

Error message

failed to create initial database session: %w

What it means

NewSessionProxy validates retry params, then calls proxy.connect(ctx) to establish the first database session; any failure there (auth selection failure, CreateDBSession error, Ping failure) is wrapped as "failed to create initial database session". Callers like Run, NewLockManager, and SessionProxyFromConfig receive this when the controller cannot establish its initial DB connection at startup.

Source

Thrown at util/sqldb/session.go:121

	if proxy.maxRetries == 0 {
		proxy.maxRetries = 5
	}
	if proxy.baseDelay == 0 {
		proxy.baseDelay = 100 * time.Millisecond
	}
	if proxy.maxDelay == 0 {
		proxy.maxDelay = 30 * time.Second
	}

	// just trying to account for float funkiness
	// a value between 0 and 1 is (almost) always non-sensical, but we allow it
	if proxy.retryMultiple <= 0.000000001 {
		proxy.retryMultiple = 1.0
	}

	if err := proxy.connect(ctx); err != nil {
		return nil, fmt.Errorf("failed to create initial database session: %w", err)
	}

	return proxy, nil
}

// NewSessionProxyFromSession creates a SessionProxy from an existing session with credentials
func NewSessionProxyFromSession(sess db.Session, dbConfig *config.DBConfig, username, password string) *SessionProxy {
	return &SessionProxy{
		sess:          sess,
		dbConfig:      dbConfig,
		username:      username,
		password:      password,
		dbType:        dbTypeFromConfig(dbConfig),
		maxRetries:    5,
		baseDelay:     100 * time.Millisecond,
		maxDelay:      30 * time.Second,
		retryMultiple: 2.0,
	}

View on GitHub (pinned to 35bff19146)

Solutions

1. Unwrap the %w cause to find the real failure (DSN error, connection refused, auth rejected, ping timeout).
2. Verify the DB host:port is reachable from the controller pod (kubectl exec + nc/psql test) and DNS resolves.
3. Check the K8s secret referenced by the persistence config contains correct username/password, or that direct username/password are both set.
4. Confirm persistence config completeness: either kubectlConfig+namespace+dbConfig or username+password+dbConfig must all be present.
5. If the DB was temporarily down, restart the controller after the database is healthy — this error occurs only at initial creation, before the reconnect logic is active.

Example fix

// before (config missing credentials path)
persistence: {postgresql: {host: pg}}  // no secret/username
// after
persistence:
  postgresql:
    host: pg
    userNameSecret: {name: argo-pg-secret, key: username}
    passwordSecret: {name: argo-pg-secret, key: password}

func cfgOK(c SessionProxyConfig) bool {
    hasKube := c.KubectlConfig != nil && c.Namespace != ""
    hasCreds := c.Username != "" && c.Password != ""
    return (hasKube || hasCreds) && !reflect.DeepEqual(c.DBConfig, config.DBConfig{})
}

proxy, err := NewSessionProxy(ctx, cfg)
if err != nil {
    var outer = err
    for unwrapped := errors.Unwrap(err); unwrapped != nil; unwrapped = errors.Unwrap(unwrapped) {
        outer = unwrapped
    }
    log.Fatalf("initial DB session failed (cause: %v) — check DB host, creds, network", outer)
}

When it happens

Trigger: NewSessionProxy fails because connect() errors: neither kubectl-config/namespace nor username/password auth combos are satisfied (insufficient auth info), CreateDBSession/CreateDBSessionWithCreds fail (bad DSN, unreachable host, wrong credentials), or sess.Ping() fails.

Common situations: Database host misconfigured or down when the controller starts; wrong Postgres/MySQL credentials in the K8s secret; network policy/Security Group blocking port 5432/3306; the persistence config partially filled so no auth branch matches; DNS resolution failure in-cluster.

Related errors

• no databases are configured
• failed to open traced postgres connection: %w
• failed to create mysql connector: %w
• failed to hydrate workflow: %w
• failed to archive workflow: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/27fae8cf7a6c7076. Report an issue: GitHub.