argoproj/argo-workflows · critical

was unable to create database connection

Error message

was unable to create database connection

What it means

NewSyncServer in server/sync/sync_server.go registers providers for semaphore/mutex synchronization. When the sync config has EnableAPI set, it creates a database-backed SessionProxy; if that proxy cannot be established (nil), the server panics because the DATABASE sync provider cannot work without a DB connection. This is a fail-fast at argo-server startup so misconfiguration is caught immediately.

Source

Thrown at server/sync/sync_server.go:38

	updateSyncLimit(ctx context.Context, req *syncpkg.UpdateSyncLimitRequest) (*syncpkg.SyncLimitResponse, error)
	deleteSyncLimit(ctx context.Context, req *syncpkg.DeleteSyncLimitRequest) (*syncpkg.DeleteSyncLimitResponse, error)
}

type syncServer struct {
	providers map[syncpkg.SyncConfigType]ConfigProvider
}

func NewSyncServer(ctx context.Context, kubectlConfig kubernetes.Interface, namespace string, syncConfig *config.SyncConfig) syncpkg.SyncServiceServer {
	server := &syncServer{
		providers: make(map[syncpkg.SyncConfigType]ConfigProvider),
	}

	server.providers[syncpkg.SyncConfigType_CONFIGMAP] = &configMapSyncProvider{}

	if syncConfig != nil && syncConfig.EnableAPI {
		sessionProxy := syncdb.SessionProxyFromConfig(ctx, kubectlConfig, namespace, syncConfig)
		if sessionProxy == nil {
			panic("was unable to create database connection")
		}
		server.providers[syncpkg.SyncConfigType_DATABASE] = &dbSyncProvider{db: syncdb.NewSyncQueries(sessionProxy, syncdb.ConfigFromConfig(syncConfig))}
	}

	return server
}

func (s *syncServer) CreateSyncLimit(ctx context.Context, req *syncpkg.CreateSyncLimitRequest) (*syncpkg.SyncLimitResponse, error) {
	if req.Limit <= 0 {
		return nil, sutils.ToStatusError(fmt.Errorf("limit must be greater than zero"), codes.InvalidArgument)
	}

	provider, ok := s.providers[req.Type]
	if !ok {
		return nil, sutils.ToStatusError(fmt.Errorf("unsupported sync config type: %s", req.Type), codes.InvalidArgument)
	}
	return provider.createSyncLimit(ctx, req)
}

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the controller configmap syncConfig block: correct database host, port, user, password, and tableName
2. Confirm the database is reachable from the argo-server pod (kubectl exec + connection test)
3. If you do not need database-backed sync, remove enableAPI: true from the sync configuration
4. Check argo-server logs for the underlying DB error printed before the panic
5. Restart argo-server after the database becomes healthy

Example fix

# before
syncConfig:
  enableAPI: true
# after
syncConfig:
  enableAPI: true
  persistence:  # ensure DB settings exist / match your database
    host: postgres
    port: 5432
    database: argo
    userNameSecret:
      name: argo-postgres-config
      key: username
    passwordSecret:
      name: argo-postgres-config
      key: password

cfg := getSyncConfig()
if cfg != nil && cfg.EnableAPI {
    if cfg.DatabaseHost == "" {
        return errors.New("syncConfig.enableAPI requires database settings")
    }
    if err := testDBConnection(cfg); err != nil {
        return fmt.Errorf("database unreachable: %w", err)
    }
}

When it happens

Trigger: Starting `argo server` with syncConfig.EnableAPI=true while the database (Postgres/MySQL) is unreachable, misconfigured (bad host/credentials), or SessionProxyFromConfig fails to open a session and returns nil.

Common situations: Enabling the sync API without configuring persistence in workflow-controller-configmap; DB pod not yet ready during server startup; wrong DB host/port/credentials; network policy blocking the server from the database.

Related errors

• Artifact driver validation failed: The following artifact dr
• unsupported db type %s
• failure to create dynamic client: %w
• must specify at least one auth mode
• failed to compile sso.filterGroupRegex: %s %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/2d1e1d20864abc5e. Report an issue: GitHub.