argoproj/argo-workflows · critical

failed to open traced postgres connection: %w

Error message

failed to open traced postgres connection: %w

What it means

This error wraps the failure from otelsql.Open when opening a traced PostgreSQL connection during database session creation in Argo Workflows. otelsql.Open is a pass-through to database/sql's sql.Open plus OpenTelemetry instrumentation; it fails when the driver cannot be found or the DSN/connection string is malformed (sql.Open does not dial yet, so this is usually a DSN/config problem, not a network problem). The underlying driver error is preserved via %w so it can be inspected with errors.Unwrap or errors.As.

Source

Thrown at util/sqldb/sqldb.go:245

	// Build PostgreSQL DSN using url.URL for safe percent-encoding of credentials
	connURL := url.URL{
		Scheme: "postgres",
		User:   url.UserPassword(username, password),
		Host:   cfg.GetHostname(),
		Path:   cfg.Database,
	}
	query := url.Values{}
	query.Set("sslmode", postgresSSLMode(cfg))
	// connect_timeout limits connection setup (dial + handshake) to ensure fast failure if the DB is unreachable.
	// lib/pq resets this deadline afterward, leaving subsequent queries unaffected.
	query.Set("connect_timeout", strconv.Itoa(int(connectTimeout.Seconds())))
	connURL.RawQuery = query.Encode()
	dsn := connURL.String()

	// Create traced *sql.DB using otelsql
	sqlDB, err := otelsql.Open("postgres", dsn, otelSQLOptions(semconv.DBSystemNamePostgreSQL, cfg.Database)...)
	if err != nil {
		return nil, fmt.Errorf("failed to open traced postgres connection: %w", err)
	}
	return newPostgresSession(sqlDB, persistPool)
}

// createMySQLDBSessionWithCreds creates MySQL DB session with direct credentials
// buildMySQLConfig constructs the mysql.Config (DSN inputs) for a MySQL session,
// using mysql.Config to safely handle special characters in credentials and
// configuring the connection-establishment (dial) timeout.
//
// Start from mysql.NewConfig() rather than a struct literal so the driver
// defaults are applied — most importantly Loc: time.UTC. When the session was
// opened from a DSN string, ParseDSN restored those defaults; NewConnector
// consumes the config directly, so a bare literal would leave Loc nil and
// panic ("missing Location in call to Time.In") on the first time.Time written.
func buildMySQLConfig(cfg *config.MySQLConfig, username, password string, connectTimeout time.Duration) (*mysql.Config, error) {
	mysqlCfg := mysql.NewConfig()
	mysqlCfg.User = username
	mysqlCfg.Passwd = password

View on GitHub (pinned to 35bff19146)

Solutions

1. Read the wrapped cause with errors.Unwrap/`%v` output and fix the reported DSN syntax problem in the Postgres config (host, port, dbname, connection options).
2. Verify the postgres connection string builds as a valid URL — special characters in username/password must be URL-escaped (e.g. use url.QueryEscape or pass credentials via secrets handling that escapes them).
3. Ensure the postgres driver is registered in your build (blank import _ "github.com/lib/pq") if building custom binaries.
4. Confirm controller --database-configuration flags / configmap keys match the documented PostgresConfig schema.

Example fix

// before (config with raw special chars in password)
postgresql:
  host: db
  password: p@ss:w0rd   # parsed into DSN unescaped -> parse error
// after
postgresql:
  host: db
  password: p%40ss%3Aw0rd   # or source from secret, escaped when building DSN

u, err := url.Parse(cfg.Host)
if err != nil || cfg.Host == "" { return fmt.Errorf("invalid postgres host/dsn: %w", err) }
// also ensure password chars are URL-escaped before building the session
_ = url.QueryEscape(password)

session, err := CreateDBSessionWithCreds(ctx)
if err != nil {
    var derr error
    if errors.As(err, &derr) { /* inspect unwrapped driver/DSN error */ }
    logger.Error(ctx, "db session creation failed", err)
    return err
}

When it happens

Trigger: CreateDBSessionWithCreds or createPostGresDBSession is called with a PostgresConfig whose DSN is invalid (bad scheme, unparseable host/port or query parameters), or the 'postgres' database/sql driver failed to register (blank import of the pq driver missing at binary build time).

Common situations: Misconfigured database section of the workflow-controller configmap (typos in host, port, or connection options); malformed URL-encoded characters in the password embedded into the DSN; custom builds that dropped the pq driver import; upgrading Argo with a changed connection-string format.

Related errors

• failed to create initial database session: %w
• no databases are configured
• only one of azureToken or awsRDSToken may be enabled, not bo
• SSL must be enabled (ssl: true) when using AWS RDS IAM authe
• failed to create mysql connector: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/c63a8ab61e878fff. Report an issue: GitHub.