argoproj/argo-workflows · critical

failed to load AWS config: %w

Error message

failed to load AWS config: %w

What it means

awsRDSConnector.Connect builds an IAM-authenticated Postgres connection by first loading the default AWS SDK v2 config (credentials, region). If awsconfig.LoadDefaultConfig cannot resolve a usable configuration, Connect wraps the SDK error with "failed to load AWS config". This is a startup/config-resolution failure, not a database failure.

Source

Thrown at util/sqldb/aws_rds_auth.go:29

	"github.com/lib/pq"
)

type awsRDSConnector struct {
	dsn      string
	endpoint string
	username string
	region   string
}

func (c *awsRDSConnector) Connect(ctx context.Context) (driver.Conn, error) {
	opts := []func(*awsconfig.LoadOptions) error{}
	if c.region != "" {
		opts = append(opts, awsconfig.WithRegion(c.region))
	}

	awsCfg, err := awsconfig.LoadDefaultConfig(ctx, opts...)
	if err != nil {
		return nil, fmt.Errorf("failed to load AWS config: %w", err)
	}

	token, err := auth.BuildAuthToken(ctx, c.endpoint, awsCfg.Region, c.username, awsCfg.Credentials)
	if err != nil {
		return nil, fmt.Errorf("failed to build RDS auth token: %w", err)
	}

	// Escape single quotes in token for safe DSN interpolation
	escapedToken := strings.ReplaceAll(token, "'", "\\'")

	dsnWithPassword := fmt.Sprintf("%s password='%s'", c.dsn, escapedToken)

	return pq.Driver{}.Open(dsnWithPassword)
}

func (c *awsRDSConnector) Driver() driver.Driver {
	return pq.Driver{}
}

View on GitHub (pinned to 35bff19146)

Solutions

1. Ensure AWS credentials are available in the environment: attach an IRSA role to the workflow-controller/service pod, or set AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, or mount a valid ~/.aws/credentials.
2. If running in-cluster, verify access to EC2 IMDS or that AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set for ECS; disable IMDS requirement by setting explicit env credentials.
3. Check AWS_PROFILE and AWS_REGION / the connector's configured region for typos and validity.
4. Validate the shared AWS config/credentials files parse correctly (AWS_SDK_LOAD_CONFIG path, INI syntax).
5. Inspect the wrapped SDK error (the %w cause) for the exact resolution chain failure (e.g. 'failed to refresh cached credentials').

Example fix

// before (pod without credentials)
# pod spec without service account
// after
# deploy.yaml
spec:
  template:
    spec:
      serviceAccountName: argo-server  # annotated with RDS IAM role (IRSA)

cfg, err := awsconfig.LoadDefaultConfig(ctx)
if err != nil || cfg.Credentials == nil {
    return fmt.Errorf("AWS config/credentials unavailable before connecting: %w", err)
}

var connErr * AWSResolutionError
if err := openDB(); err != nil {
    if strings.Contains(err.Error(), "failed to load AWS config") {
        // do not retry blindly: fix credentials/env first
        log.Fatal(err)
    }
}

When it happens

Trigger: Calling Connect on awsRDSConnector when LoadDefaultConfig fails: no AWS credentials resolvable from env vars/shared config file/IMDS/ECS/IRSA, malformed AWS_SDK_LOAD_CONFIG file, invalid profile specified via AWS_PROFILE, or unparseable regional STS endpoints.

Common situations: Argo Workflows pod has no IRSA annotation or node role lacks access to the IMDS; running locally without ~/.aws/credentials or AWS_ACCESS_KEY_ID set; a typo'd AWS_PROFILE pointing at a non-existent profile; shared config file with invalid syntax.

Related errors

• failed to obtain a credential: %w
• invalid expiry date in Kubeconfig. %w
• failed to build RDS auth token: %w
• insufficient authentication information provided
• only one of azureToken or awsRDSToken may be enabled, not bo

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/6e649288d0d0decf. Report an issue: GitHub.