argoproj/argo-workflows · critical

failed to build RDS auth token: %w

Error message

failed to build RDS auth token: %w

What it means

After loading AWS config, awsRDSConnector.Connect calls auth.BuildAuthToken to generate a presigned RDS IAM authentication token for the DB endpoint/username. If token generation fails (typically because the AWS credentials themselves cannot sign requests, or the region is empty/invalid), Connect wraps the error with "failed to build RDS auth token".

Source

Thrown at util/sqldb/aws_rds_auth.go:34

	endpoint string
	username string
	region   string
}

func (c *awsRDSConnector) Connect(ctx context.Context) (driver.Conn, error) {
	opts := []func(*awsconfig.LoadOptions) error{}
	if c.region != "" {
		opts = append(opts, awsconfig.WithRegion(c.region))
	}

	awsCfg, err := awsconfig.LoadDefaultConfig(ctx, opts...)
	if err != nil {
		return nil, fmt.Errorf("failed to load AWS config: %w", err)
	}

	token, err := auth.BuildAuthToken(ctx, c.endpoint, awsCfg.Region, c.username, awsCfg.Credentials)
	if err != nil {
		return nil, fmt.Errorf("failed to build RDS auth token: %w", err)
	}

	// Escape single quotes in token for safe DSN interpolation
	escapedToken := strings.ReplaceAll(token, "'", "\\'")

	dsnWithPassword := fmt.Sprintf("%s password='%s'", c.dsn, escapedToken)

	return pq.Driver{}.Open(dsnWithPassword)
}

func (c *awsRDSConnector) Driver() driver.Driver {
	return pq.Driver{}
}

View on GitHub (pinned to 35bff19146)

Solutions

1. Verify valid, refreshable AWS credentials are present (IRSA role, instance profile, or fresh env credentials) — re-check with `aws sts get-caller-identity` from the same environment.
2. Ensure a region is resolvable: set the connector's region field, AWS_REGION/AWS_DEFAULT_REGION env, or region in the AWS config file.
3. Confirm the DB endpoint/username match an RDS instance with IAM DB authentication enabled and the correct region.
4. Check host clock skew (NTP) — SigV4 signing fails with large drift.
5. Read the wrapped %w error: credential refresh errors vs signing errors point at different fixes.

Example fix

// before
awsCfg, err := awsconfig.LoadDefaultConfig(ctx) // no region anywhere
// after
awsCfg, err := awsconfig.LoadDefaultConfig(ctx, awsconfig.WithRegion("us-east-1"))

if awsCfg.Region == "" {
    return fmt.Errorf("AWS region must be set for RDS IAM auth")
}

if err != nil {
    var cre err := // inspect wrapped cause
    if strings.Contains(err.Error(), "failed to build RDS auth token") {
        logger.Error("RDS IAM token build failed; check credentials/region", err)
        return fmt.Errorf("rds auth: %w", err)
    }
}

When it happens

Trigger: auth.BuildAuthToken returns an error when awsCfg.Credentials fails retrieval/refresh at signing time, awsCfg.Region is empty, or the signing operation fails; the caller then wraps it as "failed to build RDS auth token".

Common situations: Credentials resolved at config-load time have since expired and cannot refresh (expired session tokens in IRSA/IMDS); region resolved to empty string because no region was configured anywhere; clock skew on the host breaking SigV4 signing; RDS endpoint not in the region of the credentials.

Related errors

• failed to load AWS config: %w
• --client-certificate and --client-key must be provided toget
• AuthSupplier cannot be empty when connecting to Argo Server
• must specify at least one auth mode
• failed to marshall claims: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/26017356c5f7d891. Report an issue: GitHub.